Aug 04 17:59:59.907539 ip-10-0-133-187 systemd[1]: Starting Kubernetes Kubelet...
Aug 04 18:00:00.321611 ip-10-0-133-187 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 18:00:00.321611 ip-10-0-133-187 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 04 18:00:00.321611 ip-10-0-133-187 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 18:00:00.321611 ip-10-0-133-187 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 04 18:00:00.321611 ip-10-0-133-187 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 18:00:00.322514 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.321678    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 04 18:00:00.325786 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325767    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:00.325786 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325786    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325790    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325793    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325797    2576 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325800    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325803    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325805    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325808    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325811    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325814    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325816    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325819    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325822    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325824    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325827    2576 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325830    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325833    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325835    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325838    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325840    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325843    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:00.325859 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325846    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325849    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325851    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325854    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325857    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325859    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325862    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325865    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325868    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325870    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325873    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325876    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325879    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325881    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325884    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325886    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325889    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325892    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325894    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325897    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:00.326369 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325900    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325902    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325905    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325908    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325910    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325912    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325915    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325918    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325920    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325923    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325927    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325931    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325934    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325936    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325940    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325943    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325946    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325949    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325952    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:00.326856 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325954    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325957    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325959    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325962    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325965    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325968    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325970    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325973    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325978    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325983    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325986    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325989    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325991    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325994    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.325997    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326000    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326003    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326005    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326007    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:00.327325 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326011    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326013    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326016    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326019    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326021    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326024    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326026    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326029    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326033    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326035    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326038    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326041    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326043    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326046    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326048    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326051    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326053    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326056    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326060    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:00.327772 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326767    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326774    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326778    2576 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326781    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326784    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326787    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326791    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326793    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326796    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326799    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326801    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326804    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326806    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326809    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326812    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326815    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326817    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326820    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326824    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326828    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:00.328250 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326831    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326834    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326837    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326840    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326842    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326845    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326847    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326850    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326853    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326857    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326860    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326863    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326867    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326870    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326872    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326875    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326878    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326880    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326882    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:00.328747 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326885    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326887    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326890    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326893    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326895    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326898    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326900    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326903    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326906    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326909    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326912    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326914    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326917    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326920    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326923    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326925    2576 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326928    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326930    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326932    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326935    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:00.329215 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326937    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326940    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326942    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326945    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326948    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326951    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326954    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326956    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326960    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326964    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326968    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326971    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326973    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326976    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326979    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326981    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326984    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326987    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326989    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:00.329706 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326992    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326995    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.326998    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327001    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327003    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327006    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327009    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327012    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327015    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327017    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327020    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327022    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327025    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327028    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327030    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327033    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327035    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327038    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327040    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327043    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:00.330174 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327045    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327417    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327422    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327425    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327428    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327431    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327433    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327436    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327439    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327441    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327444    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327447    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327450    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327453    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327456    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327459    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327462    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327464    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327469    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327472    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:00.330661 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327475    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327477    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327480    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327483    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327486    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327490    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327493    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327496    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327500    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327503    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327505    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327509    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327512    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327515    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327518    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327520    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327523    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327525    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327528    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:00.331160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327530    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327534    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327536    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327539    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327541    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327544    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327546    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327549    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327553    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327556    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327560    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327564    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327567    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327570    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327572    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327575    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327578    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327580    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327583    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:00.331645 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327585    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327588    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327591    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327593    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327596    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327598    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327602    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327605    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327607    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327610    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327612    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327615    2576 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327617    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327620    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327623    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327625    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327627    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327630    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327632    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327635    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:00.332106 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327637    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327640    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327643    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327645    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327649    2576 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327652    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327654    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327657    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327659    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327663    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327667    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327669    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327673    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327675    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327678    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327680    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327683    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327685    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327688    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327690    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:00.332658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327693    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.327696    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327759    2576 flags.go:64] FLAG: --address="0.0.0.0"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327766    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327772    2576 flags.go:64] FLAG: --anonymous-auth="true"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327777    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327781    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327784    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327788    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327792    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327796    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327799    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327802    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327806    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327808    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327812    2576 flags.go:64] FLAG: --cgroup-root=""
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327817    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327820    2576 flags.go:64] FLAG: --client-ca-file=""
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327823    2576 flags.go:64] FLAG: --cloud-provider="external"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327826    2576 flags.go:64] FLAG: --cluster-dns="[]"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327830    2576 flags.go:64] FLAG: --cluster-domain=""
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327833    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327836    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327839    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 04 18:00:00.333160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327843    2576 flags.go:64] FLAG: --container-log-max-files="5"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327851    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327854    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327857    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327861    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327864    2576 flags.go:64] FLAG: --contention-profiling="false"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327867    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327870    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327873    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327876    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327880    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327883    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327886    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327889    2576 flags.go:64] FLAG: --enable-load-reader="false"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327891    2576 flags.go:64] FLAG: --enable-server="true"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327894    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327899    2576 flags.go:64] FLAG: --event-burst="100"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327902    2576 flags.go:64] FLAG: --event-qps="50"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327905    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327908    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327911    2576 flags.go:64] FLAG: --eviction-hard=""
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327914    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327917    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327921    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327926    2576 flags.go:64] FLAG: --eviction-soft=""
Aug 04 18:00:00.333750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327930    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327933    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327936    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327939    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327942    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327944    2576 flags.go:64] FLAG: --fail-swap-on="true"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327947    2576 flags.go:64] FLAG: --feature-gates=""
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327951    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327954    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327957    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327960    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327963    2576 flags.go:64] FLAG: --healthz-port="10248"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327966    2576 flags.go:64] FLAG: --help="false"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327969    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327972    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327975    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327977    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327981    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327984    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327987    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327990    2576 flags.go:64] FLAG: --image-service-endpoint=""
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327992    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327995    2576 flags.go:64] FLAG: --kube-api-burst="100"
Aug 04 18:00:00.334385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.327999    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328002    2576 flags.go:64] FLAG: --kube-api-qps="50"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328004    2576 flags.go:64] FLAG: --kube-reserved=""
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328007    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328010    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328013    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328016    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328018    2576 flags.go:64] FLAG: --lock-file=""
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328024    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328028    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328031    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328036    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328039    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328042    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328045    2576 flags.go:64] FLAG: --logging-format="text"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328048    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328051    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328054    2576 flags.go:64] FLAG: --manifest-url=""
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328056    2576 flags.go:64] FLAG: --manifest-url-header=""
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328061    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328063    2576 flags.go:64] FLAG: --max-open-files="1000000"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328067    2576 flags.go:64] FLAG: --max-pods="110"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328070    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328073    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328076    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 04 18:00:00.335011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328079    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328081    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328085    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328087    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328094    2576 flags.go:64] FLAG: --node-status-max-images="50"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328097    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328100    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328103    2576 flags.go:64] FLAG: --pod-cidr=""
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328118    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328124    2576 flags.go:64] FLAG: --pod-manifest-path=""
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328126    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328130    2576 flags.go:64] FLAG: --pods-per-core="0"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328133    2576 flags.go:64] FLAG: --port="10250"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328136    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328139    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01d854333a2ccb215"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328143    2576 flags.go:64] FLAG: --qos-reserved=""
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328147    2576 flags.go:64] FLAG: --read-only-port="10255"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328150    2576 flags.go:64] FLAG: --register-node="true"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328153    2576 flags.go:64] FLAG: --register-with-taints=""
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328157    2576 flags.go:64] FLAG: --registry-burst="10"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328160    2576 flags.go:64] FLAG: --registry-qps="5"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328162    2576 flags.go:64] FLAG: --reserved-cpus=""
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328165    2576 flags.go:64] FLAG: --reserved-memory=""
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328169    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328172    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 04 18:00:00.335670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328175    2576 flags.go:64] FLAG: --rotate-certificates="false"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328178    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328181    2576 flags.go:64] FLAG: --runonce="false"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328184    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328187    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328190    2576 flags.go:64] FLAG: --seccomp-default="false"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328192    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328195    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328198    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328202    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328205    2576 flags.go:64] FLAG: --storage-driver-password="root"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328208    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328211    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328213    2576 flags.go:64] FLAG: --storage-driver-user="root"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328216    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328219    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328222    2576 flags.go:64] FLAG: --system-cgroups=""
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328225    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328230    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328233    2576 flags.go:64] FLAG: --tls-cert-file=""
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328236    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328240    2576 flags.go:64] FLAG: --tls-min-version=""
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328244    2576 flags.go:64] FLAG: --tls-private-key-file=""
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328248    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328251    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 04 18:00:00.336293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328254    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328257    2576 flags.go:64] FLAG: --v="2"
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328262    2576 flags.go:64] FLAG: --version="false"
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328266    2576 flags.go:64] FLAG: --vmodule=""
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328269    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.328272    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328389    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328393    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328396    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328399    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328401    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328404    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328407    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328409    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328412    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328414    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328417    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328420    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328422    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328425    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328427    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:00.336905 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328430    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328432    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328435    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328437    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328440    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328442    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328445    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328448    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328452    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328455    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328458    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328460    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328463    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328467    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328471    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328474    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328476    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328479    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328481    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:00.337459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328484    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328488    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328491    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328494    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328497    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328500    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328502    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328505    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328507    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328510    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328512    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328515    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328517    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328520    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328522    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328524    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328527    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328530    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328532    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:00.337966 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328535    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328537    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328542    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328545    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328547    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328550    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328553    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328556    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328559    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328561    2576 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328564    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328566    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328569    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328571    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328574    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328578    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328580    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328583    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328585    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328588    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:00.338453 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328590    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328592    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328595    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328597    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328601    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328605    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328607    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328610    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328613    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328616    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328619    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328621    2576 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328624    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328627    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328631    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328634    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328637    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328639    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328642    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:00.338944 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328644    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328647    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328650    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328652    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328654    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328657    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.328659    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.329325    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.336419    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.336443    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.337781    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.337790    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337859    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337865    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337869    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:00.339434 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337873    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337876    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337879    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337884    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337887    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337890    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337893    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337895    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337898    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337901    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337903    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337906    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337909    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337911    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337913    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337916    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337919    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337921    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337924    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:00.339812 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337926    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337929    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337931    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337934    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337936    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337941    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337944    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337954    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337957    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337961    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337963    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337966    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337969    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337972    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337975    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337978    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337980    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337983    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337986    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337989    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:00.340280 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337991    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337994    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337996    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.337999    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338001    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338004    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338007    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338009    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338012    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338015    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338019    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338022    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338025    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338027    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338030    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338033    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338035    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338038    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338042    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:00.340758 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338045    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338054    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338057    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338059    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338062    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338064    2576 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338067    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338070    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338072    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338075    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338077    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338080    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338083    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338085    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338088    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338091    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338093    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338096    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338098    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338101    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:00.341296 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338104    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338121    2576 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338125    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338127    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338130    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338132    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338135    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338137    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338140    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338143    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338147    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338151    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338154    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338157    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338166    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338169    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338172    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:00.341813 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338175    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.338180    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338323    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338328    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338331    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338334    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338337    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338339    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338342    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338345    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338347    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338350    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338353    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338355    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338358    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:00.342261 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338360    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338363    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338367    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338370    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338373    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338376    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338378    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338381    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338383    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338386    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338389    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338391    2576 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338394    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338396    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338399    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338407    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338411    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338414    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:00.342633 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338417    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338421    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338424    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338428    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338431    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338433    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338436    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338438    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338441    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338443    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338446    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338448    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338450    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338453    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338455    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338458    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338461    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338464    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338466    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:00.343075 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338469    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338472    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338474    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338476    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338479    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338482    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338485    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338488    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338491    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338494    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338496    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338523    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338527    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338530    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338533    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338536    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338539    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338541    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338544    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:00.343580 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338547    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338549    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338552    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338554    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338557    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338559    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338562    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338564    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338566    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338569    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338572    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338574    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338577    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338579    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338582    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338584    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338587    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338590    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338593    2576 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338595    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:00.344038 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338598    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:00.344662 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338601    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:00.344662 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338603    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:00.344662 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338606    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:00.344662 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338608    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:00.344662 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338617    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:00.344662 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338619    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:00.344662 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338622    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:00.344662 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338624    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:00.344662 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:00.338627    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:00.344662 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.338632    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 18:00:00.344662 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.338791    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 04 18:00:00.344662 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.341087    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 04 18:00:00.344662 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.343217    2576 server.go:1019] "Starting client certificate rotation"
Aug 04 18:00:00.344662 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.343335    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 18:00:00.345015 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.344158    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 18:00:00.345637 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.345621    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 18:00:00.348358 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.348261    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 18:00:00.359962 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.359941    2576 log.go:25] "Validated CRI v1 runtime API"
Aug 04 18:00:00.362559 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.362542    2576 log.go:25] "Validated CRI v1 image API"
Aug 04 18:00:00.363306 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.363292    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 04 18:00:00.365601 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.365578    2576 fs.go:135] Filesystem UUIDs: map[78015994-9a4e-4e37-bad3-6aac377a1514:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7d2ca7dc-d9f0-4e9a-86a5-f30fbc812c31:/dev/nvme0n1p4]
Aug 04 18:00:00.365662 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.365602    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 04 18:00:00.371487 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.371379    2576 manager.go:216] Machine: {Timestamp:2026-08-04 18:00:00.369993003 +0000 UTC m=+0.345824691 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102404 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21d57e5f8f8a869eb620a923be1565 SystemUUID:ec21d57e-5f8f-8a86-9eb6-20a923be1565 BootID:997c23db-7a7f-48e8-ade4-b1c8f7434a0d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5e:0e:d7:4d:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5e:0e:d7:4d:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:88:93:11:c8:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 04 18:00:00.371487 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.371481    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 04 18:00:00.371608 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.371562    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 04 18:00:00.374857 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.374830    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 04 18:00:00.375041 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.374860    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-187.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 04 18:00:00.376276 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.376264    2576 topology_manager.go:138] "Creating topology manager with none policy"
Aug 04 18:00:00.376333 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.376283    2576 container_manager_linux.go:309] "Creating device plugin manager"
Aug 04 18:00:00.376333 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.376295    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 18:00:00.377024 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.377011    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 18:00:00.379329 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.379316    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 04 18:00:00.379388 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.379316    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 18:00:00.383876 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.383862    2576 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 18:00:00.384017 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.384006    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 04 18:00:00.384072 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.384044    2576 kubelet.go:486] "Attempting to sync node with API server"
Aug 04 18:00:00.384072 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.384066    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 04 18:00:00.384183 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.384087    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 04 18:00:00.384183 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.384103    2576 kubelet.go:398] "Adding apiserver pod source"
Aug 04 18:00:00.384183 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.384132    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 04 18:00:00.384585 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.384571    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 18:00:00.384652 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.384595    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 18:00:00.386869 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.386850    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 04 18:00:00.390317 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.390286    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 04 18:00:00.390416 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.390345    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 04 18:00:00.392891 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.392868    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 04 18:00:00.392953 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.392896    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 04 18:00:00.392953 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.392903    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 04 18:00:00.392953 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.392908    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 04 18:00:00.392953 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.392917    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 04 18:00:00.392953 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.392923    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 04 18:00:00.392953 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.392938    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 04 18:00:00.392953 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.392945    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 04 18:00:00.392953 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.392952    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 04 18:00:00.393183 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.392958    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 04 18:00:00.393183 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.392968    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 04 18:00:00.393183 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.392974    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 04 18:00:00.394013 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.394002    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 04 18:00:00.394013 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.394013    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 04 18:00:00.394499 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.394476    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 04 18:00:00.394542 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.394477    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 04 18:00:00.397141 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.397128    2576 server.go:1268] "Started kubelet"
Aug 04 18:00:00.397240 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.397203    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 04 18:00:00.397301 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.397216    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 04 18:00:00.397352 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.397304    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 04 18:00:00.398092 ip-10-0-133-187 systemd[1]: Started Kubernetes Kubelet.
Aug 04 18:00:00.398428 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.398409    2576 server.go:310] "Adding debug handlers to kubelet server"
Aug 04 18:00:00.398918 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.398875    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 04 18:00:00.401217 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.401196    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wtc6f"
Aug 04 18:00:00.403164 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.403140    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 04 18:00:00.403474 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.403460    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-187.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 04 18:00:00.404642 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.403488    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-187.ec2.internal.18c8ac077bf9257c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-187.ec2.internal,UID:ip-10-0-133-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-187.ec2.internal,},FirstTimestamp:2026-08-04 18:00:00.397075836 +0000 UTC m=+0.372907526,LastTimestamp:2026-08-04 18:00:00.397075836 +0000 UTC m=+0.372907526,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-187.ec2.internal,}"
Aug 04 18:00:00.405660 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.405645    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 04 18:00:00.405660 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.405650    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 04 18:00:00.406197 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.406165    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found"
Aug 04 18:00:00.406459 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.406439    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 04 18:00:00.406459 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.406461    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 04 18:00:00.406593 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.406439    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 04 18:00:00.406593 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.406548    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 04 18:00:00.406593 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.406561    2576 factory.go:55] Registering systemd factory
Aug 04 18:00:00.406593 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.406571    2576 factory.go:223] Registration of the systemd container factory successfully
Aug 04 18:00:00.406593 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.406586    2576 reconstruct.go:97] "Volume reconstruction finished"
Aug 04 18:00:00.406593 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.406596    2576 reconciler.go:29] "Reconciler: start to sync state"
Aug 04 18:00:00.406842 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.406832    2576 factory.go:153] Registering CRI-O factory
Aug 04 18:00:00.406842 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.406843    2576 factory.go:223] Registration of the crio container factory successfully
Aug 04 18:00:00.406905 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.406861    2576 factory.go:103] Registering Raw factory
Aug 04 18:00:00.406905 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.406871    2576 manager.go:1196] Started watching for new ooms in manager
Aug 04 18:00:00.407491 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.407477    2576 manager.go:349] Starting recovery of all containers
Aug 04 18:00:00.413353 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.413326    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Aug 04 18:00:00.413460 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.413391    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Aug 04 18:00:00.413517 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.413479    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wtc6f"
Aug 04 18:00:00.418753 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.418733    2576 manager.go:354] Recovery completed
Aug 04 18:00:00.424209 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.424082    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:00.425972 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.425956    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:00.426042 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.425985    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:00.426042 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.425995    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:00.426588 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.426575    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 04 18:00:00.426588 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.426587    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 04 18:00:00.426675 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.426601    2576 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 18:00:00.426675 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.426631    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:00.427354 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.427340    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:00.427421 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.427369    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:00.427421 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.427385    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:00.427821 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.427808    2576 policy_none.go:49] "None policy: Start"
Aug 04 18:00:00.427863 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.427824    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 04 18:00:00.427863 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.427835    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 04 18:00:00.428852 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.428840    2576 policy_none.go:47] "Start"
Aug 04 18:00:00.429317 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.429215    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-187.ec2.internal.18c8ac077db21210  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-187.ec2.internal,UID:ip-10-0-133-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-187.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-187.ec2.internal,},FirstTimestamp:2026-08-04 18:00:00.42597224 +0000 UTC m=+0.401803928,LastTimestamp:2026-08-04 18:00:00.42597224 +0000 UTC m=+0.401803928,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-187.ec2.internal,}"
Aug 04 18:00:00.465743 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.465726    2576 manager.go:337] "Starting Device Plugin manager"
Aug 04 18:00:00.465863 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.465764    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 04 18:00:00.465863 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.465774    2576 server.go:87] "Starting device plugin registration server"
Aug 04 18:00:00.466049 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.466038    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 04 18:00:00.466102 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.466051    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 04 18:00:00.466187 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.466163    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 04 18:00:00.466257 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.466241    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 04 18:00:00.466257 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.466249    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 04 18:00:00.466781 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.466763    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 04 18:00:00.466864 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.466806    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-187.ec2.internal\" not found"
Aug 04 18:00:00.569129 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.569091    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:00.570524 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.570178    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:00.570524 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.570215    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:00.570524 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.570226    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:00.570524 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.570255    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.574363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.574299    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 04 18:00:00.575539 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.575523    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 04 18:00:00.575628 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.575560    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 04 18:00:00.575628 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.575580    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 04 18:00:00.575628 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.575613    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 04 18:00:00.577238 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.577219    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.577320 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.577244    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-187.ec2.internal\": node \"ip-10-0-133-187.ec2.internal\" not found"
Aug 04 18:00:00.577744 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.577722    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:00.622252 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.622223    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found"
Aug 04 18:00:00.678861 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.678823    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal"]
Aug 04 18:00:00.678980 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.678922    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:00.680159 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.680143    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:00.680255 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.680174    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:00.680255 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.680188    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:00.681875 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.681861    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:00.682004 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.681990    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.682043 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.682022    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:00.682791 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.682777    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:00.682863 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.682807    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:00.682863 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.682820    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:00.683511 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.683488    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:00.683616 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.683524    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:00.683616 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.683536    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:00.685425 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.685407    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.685525 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.685434    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:00.686234 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.686218    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:00.686316 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.686251    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:00.686316 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.686264    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:00.700825 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.700802    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-187.ec2.internal\" not found" node="ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.705602 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.705586    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-187.ec2.internal\" not found" node="ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.708793 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.708774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/442faadf703363fe454f012df8df1128-config\") pod \"kube-apiserver-proxy-ip-10-0-133-187.ec2.internal\" (UID: \"442faadf703363fe454f012df8df1128\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.708869 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.708802    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53a66f6347b8909039c9c2cd29191e55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal\" (UID: \"53a66f6347b8909039c9c2cd29191e55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.708869 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.708820    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53a66f6347b8909039c9c2cd29191e55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal\" (UID: \"53a66f6347b8909039c9c2cd29191e55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.722305 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.722274    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found"
Aug 04 18:00:00.809406 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.809364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53a66f6347b8909039c9c2cd29191e55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal\" (UID: \"53a66f6347b8909039c9c2cd29191e55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.809406 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.809406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/442faadf703363fe454f012df8df1128-config\") pod \"kube-apiserver-proxy-ip-10-0-133-187.ec2.internal\" (UID: \"442faadf703363fe454f012df8df1128\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.809641 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.809423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53a66f6347b8909039c9c2cd29191e55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal\" (UID: \"53a66f6347b8909039c9c2cd29191e55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.809641 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.809461    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/442faadf703363fe454f012df8df1128-config\") pod \"kube-apiserver-proxy-ip-10-0-133-187.ec2.internal\" (UID: \"442faadf703363fe454f012df8df1128\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.809641 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.809470    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53a66f6347b8909039c9c2cd29191e55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal\" (UID: \"53a66f6347b8909039c9c2cd29191e55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.809641 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:00.809501    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53a66f6347b8909039c9c2cd29191e55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal\" (UID: \"53a66f6347b8909039c9c2cd29191e55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:00.822775 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.822748    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found"
Aug 04 18:00:00.923090 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:00.923006    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found"
Aug 04 18:00:01.009900 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:01.009861    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:01.010000 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:01.009910    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:01.023934 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:01.023910    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found"
Aug 04 18:00:01.124315 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:01.124253    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found"
Aug 04 18:00:01.224754 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:01.224648    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found"
Aug 04 18:00:01.325831 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:01.325788    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found"
Aug 04 18:00:01.342175 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:01.342133    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 04 18:00:01.342358 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:01.342337    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 18:00:01.420527 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:01.420489    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 17:55:01 +0000 UTC" deadline="2028-04-05 08:46:47.448420092 +0000 UTC"
Aug 04 18:00:01.420527 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:01.420519    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14630h46m46.027904567s"
Aug 04 18:00:01.426431 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:01.426412    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found"
Aug 04 18:00:01.526904 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:01.526829    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found"
Aug 04 18:00:01.574062 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:01.574039    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:01.606661 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:01.606616    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:01.618686 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:01.618663    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 18:00:01.620628 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:01.620612    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal"
Aug 04 18:00:01.629886 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:01.629864    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 18:00:01.764556 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:01.764527    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:01.853712 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:01.853683    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:02.387956 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.387908    2576 apiserver.go:52] "Watching apiserver"
Aug 04 18:00:02.401521 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.401488    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 04 18:00:02.402706 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.402683    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal","openshift-image-registry/node-ca-t69hm","openshift-multus/multus-additional-cni-plugins-zh6dz","openshift-multus/network-metrics-daemon-7ggnb","openshift-network-diagnostics/network-check-target-q5kpw","openshift-network-operator/iptables-alerter-jmm68","kube-system/global-pull-secret-syncer-6kdvz","kube-system/konnectivity-agent-9ggjx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2","openshift-cluster-node-tuning-operator/tuned-7x95l","openshift-dns/node-resolver-vs6m9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal","openshift-multus/multus-5q6q7","openshift-ovn-kubernetes/ovnkube-node-8rlhj"]
Aug 04 18:00:02.404406 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.404393    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:02.404486 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:02.404465    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:02.405697 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.405677    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.406615 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.406588    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 04 18:00:02.407180 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.407160    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.408647 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.408628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:02.408734 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:02.408695    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:02.409608 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.409585    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jc6bm\""
Aug 04 18:00:02.409735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.409604    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 04 18:00:02.409735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.409634    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 04 18:00:02.409735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.409588    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 04 18:00:02.409735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.409701    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 04 18:00:02.409965 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.409946    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:02.410041 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:02.410019    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:02.410175 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.410159    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6lmdn\""
Aug 04 18:00:02.410981 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.410964    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 04 18:00:02.411069 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.410966    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 04 18:00:02.411436 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.411422    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jmm68"
Aug 04 18:00:02.412810 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.412793    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.414206 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.414180    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.414719 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.414700    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:02.414932 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.414916    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 04 18:00:02.415015 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.414994    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f4srf\""
Aug 04 18:00:02.415079 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.415057    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:02.415581 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.415561    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t69hm"
Aug 04 18:00:02.417170 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417151    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 04 18:00:02.417247 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417151    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 04 18:00:02.417309 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417292    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.417530 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/707bcbec-504f-4e07-ae6f-6d6b4d16964e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.417567 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417541    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:02.417567 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417561    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-multus-socket-dir-parent\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.417638 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417577    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-run-k8s-cni-cncf-io\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.417638 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-var-lib-kubelet\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.417638 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417615    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f0d65e6e-91ca-476e-b30e-3da05d33616e-multus-daemon-config\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.417638 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/707bcbec-504f-4e07-ae6f-6d6b4d16964e-cnibin\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.417788 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417650    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-kubelet-config\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:02.417788 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-os-release\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.417788 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f0d65e6e-91ca-476e-b30e-3da05d33616e-cni-binary-copy\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.417788 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-run-netns\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.417788 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-hostroot\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.417788 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-run-multus-certs\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.417788 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417760    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/707bcbec-504f-4e07-ae6f-6d6b4d16964e-system-cni-dir\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.417788 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/707bcbec-504f-4e07-ae6f-6d6b4d16964e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.418021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-system-cni-dir\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.418021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-multus-cni-dir\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.418021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417845    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/707bcbec-504f-4e07-ae6f-6d6b4d16964e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.418021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2pq8\" (UniqueName: \"kubernetes.io/projected/707bcbec-504f-4e07-ae6f-6d6b4d16964e-kube-api-access-n2pq8\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.418021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417912    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-cnibin\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.418021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-var-lib-cni-multus\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.418021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpd4f\" (UniqueName: \"kubernetes.io/projected/f0d65e6e-91ca-476e-b30e-3da05d33616e-kube-api-access-kpd4f\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.418021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417970    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/707bcbec-504f-4e07-ae6f-6d6b4d16964e-os-release\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.418021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.417991    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-var-lib-cni-bin\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.418021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.418005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-multus-conf-dir\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.418021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.418020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-etc-kubernetes\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.418373 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.418038    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/707bcbec-504f-4e07-ae6f-6d6b4d16964e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.418373 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.418077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-dbus\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:02.418976 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.418962    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vs6m9"
Aug 04 18:00:02.419207 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.419172    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nccl5\""
Aug 04 18:00:02.419336 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.419320    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 04 18:00:02.420055 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.420043    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9kpct\""
Aug 04 18:00:02.420406 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.420392    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9ggjx"
Aug 04 18:00:02.420544 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.420528    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 04 18:00:02.425218 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.424950    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 04 18:00:02.425218 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.425021    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 04 18:00:02.425218 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.425045    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 04 18:00:02.425218 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.425083    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 04 18:00:02.425218 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.425204    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 04 18:00:02.425510 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.425246    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 04 18:00:02.425510 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.425261    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 04 18:00:02.425510 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.425301    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 04 18:00:02.426420 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.425894    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pjmtd\""
Aug 04 18:00:02.426922 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.426730    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:02.427515 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.427490    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-npbs9\""
Aug 04 18:00:02.427665 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.427644    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:02.429995 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.429972    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 04 18:00:02.430569 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.430545    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 04 18:00:02.430683 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.430600    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 04 18:00:02.430683 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.430629    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 04 18:00:02.430683 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.430643    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9rmkq\""
Aug 04 18:00:02.433338 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.433317    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n48gm\""
Aug 04 18:00:02.435082 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.435065    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 18:00:02.467342 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.467313    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rslkc"
Aug 04 18:00:02.486427 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.486400    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rslkc"
Aug 04 18:00:02.507455 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.507438    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 04 18:00:02.518786 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518763    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-kubelet-config\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:02.518907 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-os-release\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.518907 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518812    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-run-netns\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.518907 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-hostroot\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.518907 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518847    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-sysctl-d\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.518907 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/707bcbec-504f-4e07-ae6f-6d6b4d16964e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.518907 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518882    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-kubelet-config\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:02.518907 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518901    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-os-release\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.519157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518888    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-run-netns\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.519157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518904    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-hostroot\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.519157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518890    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-system-cni-dir\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.519157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518959    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-multus-cni-dir\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.519157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4r22\" (UniqueName: \"kubernetes.io/projected/62165010-2435-476c-bf99-75bf927b2b45-kube-api-access-x4r22\") pod \"network-metrics-daemon-7ggnb\" (UID: \"62165010-2435-476c-bf99-75bf927b2b45\") " pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:02.519157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c532b0b-be73-4f6f-be91-7ae296f21534-tmp\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.519157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r248r\" (UniqueName: \"kubernetes.io/projected/2c532b0b-be73-4f6f-be91-7ae296f21534-kube-api-access-r248r\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.519157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.518935    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-system-cni-dir\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.519157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519036    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpmm5\" (UniqueName: \"kubernetes.io/projected/22866d7a-8cbe-4c71-8ffb-507f5d4b879f-kube-api-access-kpmm5\") pod \"node-ca-t69hm\" (UID: \"22866d7a-8cbe-4c71-8ffb-507f5d4b879f\") " pod="openshift-image-registry/node-ca-t69hm"
Aug 04 18:00:02.519157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/707bcbec-504f-4e07-ae6f-6d6b4d16964e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.519157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519097    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpd4f\" (UniqueName: \"kubernetes.io/projected/f0d65e6e-91ca-476e-b30e-3da05d33616e-kube-api-access-kpd4f\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.519157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519111    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-multus-cni-dir\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.519157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519120    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-sysctl-conf\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519180    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/707bcbec-504f-4e07-ae6f-6d6b4d16964e-os-release\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519234    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-var-lib-cni-bin\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519249    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/707bcbec-504f-4e07-ae6f-6d6b4d16964e-os-release\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-multus-conf-dir\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519298    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-var-lib-cni-bin\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519340    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/707bcbec-504f-4e07-ae6f-6d6b4d16964e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519343    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-etc-kubernetes\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519380    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-etc-kubernetes\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519424    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519453    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/707bcbec-504f-4e07-ae6f-6d6b4d16964e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519453    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-multus-conf-dir\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519485    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-dbus\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-sys-fs\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519532    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-sysconfig\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30-konnectivity-ca\") pod \"konnectivity-agent-9ggjx\" (UID: \"b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30\") " pod="kube-system/konnectivity-agent-9ggjx"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519551    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-dbus\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:02.519726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g5rg\" (UniqueName: \"kubernetes.io/projected/55a0c1ed-dbff-455b-b25e-2fc6741f0876-kube-api-access-5g5rg\") pod \"iptables-alerter-jmm68\" (UID: \"55a0c1ed-dbff-455b-b25e-2fc6741f0876\") " pod="openshift-network-operator/iptables-alerter-jmm68"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-etc-openvswitch\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519602    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-log-socket\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519617    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-cni-bin\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs\") pod \"network-metrics-daemon-7ggnb\" (UID: \"62165010-2435-476c-bf99-75bf927b2b45\") " pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/821095f1-8366-4898-9b00-1656eb70c735-env-overrides\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/821095f1-8366-4898-9b00-1656eb70c735-ovnkube-script-lib\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-multus-socket-dir-parent\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519725    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-var-lib-kubelet\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f0d65e6e-91ca-476e-b30e-3da05d33616e-multus-daemon-config\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-device-dir\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519781    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-multus-socket-dir-parent\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-host\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519844    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-var-lib-kubelet\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55a0c1ed-dbff-455b-b25e-2fc6741f0876-host-slash\") pod \"iptables-alerter-jmm68\" (UID: \"55a0c1ed-dbff-455b-b25e-2fc6741f0876\") " pod="openshift-network-operator/iptables-alerter-jmm68"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-run-netns\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519962    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/821095f1-8366-4898-9b00-1656eb70c735-ovn-node-metrics-cert\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.520643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.519990    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95nwn\" (UniqueName: \"kubernetes.io/projected/924363b2-77bd-4de5-9279-8fdf18129796-kube-api-access-95nwn\") pod \"node-resolver-vs6m9\" (UID: \"924363b2-77bd-4de5-9279-8fdf18129796\") " pod="openshift-dns/node-resolver-vs6m9"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520015    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-kubelet\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520048    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22866d7a-8cbe-4c71-8ffb-507f5d4b879f-host\") pod \"node-ca-t69hm\" (UID: \"22866d7a-8cbe-4c71-8ffb-507f5d4b879f\") " pod="openshift-image-registry/node-ca-t69hm"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/707bcbec-504f-4e07-ae6f-6d6b4d16964e-cnibin\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f0d65e6e-91ca-476e-b30e-3da05d33616e-cni-binary-copy\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520141    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/707bcbec-504f-4e07-ae6f-6d6b4d16964e-cnibin\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520155    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-run-multus-certs\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ttb\" (UniqueName: \"kubernetes.io/projected/897dc98e-7462-4c2c-8579-64a0e783d074-kube-api-access-p9ttb\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/924363b2-77bd-4de5-9279-8fdf18129796-hosts-file\") pod \"node-resolver-vs6m9\" (UID: \"924363b2-77bd-4de5-9279-8fdf18129796\") " pod="openshift-dns/node-resolver-vs6m9"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520227    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-run-multus-certs\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/924363b2-77bd-4de5-9279-8fdf18129796-tmp-dir\") pod \"node-resolver-vs6m9\" (UID: \"924363b2-77bd-4de5-9279-8fdf18129796\") " pod="openshift-dns/node-resolver-vs6m9"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-run-ovn-kubernetes\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520265    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f0d65e6e-91ca-476e-b30e-3da05d33616e-multus-daemon-config\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/707bcbec-504f-4e07-ae6f-6d6b4d16964e-system-cni-dir\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520361    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520384    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/707bcbec-504f-4e07-ae6f-6d6b4d16964e-system-cni-dir\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520410    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-systemd\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.521375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520451    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-run\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-lib-modules\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30-agent-certs\") pod \"konnectivity-agent-9ggjx\" (UID: \"b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30\") " pod="kube-system/konnectivity-agent-9ggjx"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-node-log\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520563    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvrqc\" (UniqueName: \"kubernetes.io/projected/821095f1-8366-4898-9b00-1656eb70c735-kube-api-access-xvrqc\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2pq8\" (UniqueName: \"kubernetes.io/projected/707bcbec-504f-4e07-ae6f-6d6b4d16964e-kube-api-access-n2pq8\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f0d65e6e-91ca-476e-b30e-3da05d33616e-cni-binary-copy\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520620    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-cnibin\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-var-lib-cni-multus\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520667    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-sys\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55a0c1ed-dbff-455b-b25e-2fc6741f0876-iptables-alerter-script\") pod \"iptables-alerter-jmm68\" (UID: \"55a0c1ed-dbff-455b-b25e-2fc6741f0876\") " pod="openshift-network-operator/iptables-alerter-jmm68"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520706    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-cnibin\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520734    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-var-lib-cni-multus\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520747    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-run-openvswitch\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-run-ovn\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520819    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-cni-netd\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-systemd-units\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.521898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520868    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/821095f1-8366-4898-9b00-1656eb70c735-ovnkube-config\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/707bcbec-504f-4e07-ae6f-6d6b4d16964e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-tuned\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.520961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-slash\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521035    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-modprobe-d\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-kubernetes\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521079    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/707bcbec-504f-4e07-ae6f-6d6b4d16964e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521134    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-run-k8s-cni-cncf-io\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521158    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-etc-selinux\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521189    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f0d65e6e-91ca-476e-b30e-3da05d33616e-host-run-k8s-cni-cncf-io\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-var-lib-kubelet\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:02.521219    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521240    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-run-systemd\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521265    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmsxm\" (UniqueName: \"kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm\") pod \"network-check-target-q5kpw\" (UID: \"110d78d6-4775-4717-84e3-030a8c109a9b\") " pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521290    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-socket-dir\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.522426 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:02.521324    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret podName:e9274e13-87e6-4bb8-bb29-3ed6fcde80e0 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:03.021287208 +0000 UTC m=+2.997118904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret") pod "global-pull-secret-syncer-6kdvz" (UID: "e9274e13-87e6-4bb8-bb29-3ed6fcde80e0") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:02.522907 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-registration-dir\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.522907 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521414    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/707bcbec-504f-4e07-ae6f-6d6b4d16964e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.522907 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521420    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-var-lib-openvswitch\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.522907 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22866d7a-8cbe-4c71-8ffb-507f5d4b879f-serviceca\") pod \"node-ca-t69hm\" (UID: \"22866d7a-8cbe-4c71-8ffb-507f5d4b879f\") " pod="openshift-image-registry/node-ca-t69hm"
Aug 04 18:00:02.522907 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.521579    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/707bcbec-504f-4e07-ae6f-6d6b4d16964e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.526507 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.526488    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 04 18:00:02.529772 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.529751    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpd4f\" (UniqueName: \"kubernetes.io/projected/f0d65e6e-91ca-476e-b30e-3da05d33616e-kube-api-access-kpd4f\") pod \"multus-5q6q7\" (UID: \"f0d65e6e-91ca-476e-b30e-3da05d33616e\") " pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.529868 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.529754    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2pq8\" (UniqueName: \"kubernetes.io/projected/707bcbec-504f-4e07-ae6f-6d6b4d16964e-kube-api-access-n2pq8\") pod \"multus-additional-cni-plugins-zh6dz\" (UID: \"707bcbec-504f-4e07-ae6f-6d6b4d16964e\") " pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.622003 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.621968    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-device-dir\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.622003 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.621999    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-host\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.622235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55a0c1ed-dbff-455b-b25e-2fc6741f0876-host-slash\") pod \"iptables-alerter-jmm68\" (UID: \"55a0c1ed-dbff-455b-b25e-2fc6741f0876\") " pod="openshift-network-operator/iptables-alerter-jmm68"
Aug 04 18:00:02.622235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-run-netns\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.622235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622055    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/821095f1-8366-4898-9b00-1656eb70c735-ovn-node-metrics-cert\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.622235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622071    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95nwn\" (UniqueName: \"kubernetes.io/projected/924363b2-77bd-4de5-9279-8fdf18129796-kube-api-access-95nwn\") pod \"node-resolver-vs6m9\" (UID: \"924363b2-77bd-4de5-9279-8fdf18129796\") " pod="openshift-dns/node-resolver-vs6m9"
Aug 04 18:00:02.622235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622090    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-kubelet\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.622235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622096    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-device-dir\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.622235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22866d7a-8cbe-4c71-8ffb-507f5d4b879f-host\") pod \"node-ca-t69hm\" (UID: \"22866d7a-8cbe-4c71-8ffb-507f5d4b879f\") " pod="openshift-image-registry/node-ca-t69hm"
Aug 04 18:00:02.622235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622111    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55a0c1ed-dbff-455b-b25e-2fc6741f0876-host-slash\") pod \"iptables-alerter-jmm68\" (UID: \"55a0c1ed-dbff-455b-b25e-2fc6741f0876\") " pod="openshift-network-operator/iptables-alerter-jmm68"
Aug 04 18:00:02.622235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622141    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-run-netns\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.622235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622111    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-host\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.622235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622162    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22866d7a-8cbe-4c71-8ffb-507f5d4b879f-host\") pod \"node-ca-t69hm\" (UID: \"22866d7a-8cbe-4c71-8ffb-507f5d4b879f\") " pod="openshift-image-registry/node-ca-t69hm"
Aug 04 18:00:02.622235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622167    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ttb\" (UniqueName: \"kubernetes.io/projected/897dc98e-7462-4c2c-8579-64a0e783d074-kube-api-access-p9ttb\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.622235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622198    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-kubelet\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/924363b2-77bd-4de5-9279-8fdf18129796-hosts-file\") pod \"node-resolver-vs6m9\" (UID: \"924363b2-77bd-4de5-9279-8fdf18129796\") " pod="openshift-dns/node-resolver-vs6m9"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/924363b2-77bd-4de5-9279-8fdf18129796-tmp-dir\") pod \"node-resolver-vs6m9\" (UID: \"924363b2-77bd-4de5-9279-8fdf18129796\") " pod="openshift-dns/node-resolver-vs6m9"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622344    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/924363b2-77bd-4de5-9279-8fdf18129796-hosts-file\") pod \"node-resolver-vs6m9\" (UID: \"924363b2-77bd-4de5-9279-8fdf18129796\") " pod="openshift-dns/node-resolver-vs6m9"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-run-ovn-kubernetes\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622421    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622447    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-systemd\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622475    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-run-ovn-kubernetes\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622494    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-systemd\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-run\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622506    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622549    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-run\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-lib-modules\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622596    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30-agent-certs\") pod \"konnectivity-agent-9ggjx\" (UID: \"b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30\") " pod="kube-system/konnectivity-agent-9ggjx"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622553    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/924363b2-77bd-4de5-9279-8fdf18129796-tmp-dir\") pod \"node-resolver-vs6m9\" (UID: \"924363b2-77bd-4de5-9279-8fdf18129796\") " pod="openshift-dns/node-resolver-vs6m9"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622623    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-node-log\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622648    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvrqc\" (UniqueName: \"kubernetes.io/projected/821095f1-8366-4898-9b00-1656eb70c735-kube-api-access-xvrqc\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622668    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-lib-modules\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.622801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-sys\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622693    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-node-log\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622698    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-sys\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622717    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55a0c1ed-dbff-455b-b25e-2fc6741f0876-iptables-alerter-script\") pod \"iptables-alerter-jmm68\" (UID: \"55a0c1ed-dbff-455b-b25e-2fc6741f0876\") " pod="openshift-network-operator/iptables-alerter-jmm68"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-run-openvswitch\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622772    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-run-ovn\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-cni-netd\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622821    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-run-openvswitch\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-systemd-units\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622854    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-run-ovn\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622908    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-systemd-units\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622870    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-cni-netd\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622873    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/821095f1-8366-4898-9b00-1656eb70c735-ovnkube-config\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.622974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-tuned\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-slash\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-modprobe-d\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-kubernetes\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623127    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-slash\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.623785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623145    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-kubernetes\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-etc-selinux\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-var-lib-kubelet\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623218    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-run-systemd\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623242    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsxm\" (UniqueName: \"kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm\") pod \"network-check-target-q5kpw\" (UID: \"110d78d6-4775-4717-84e3-030a8c109a9b\") " pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623248    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-modprobe-d\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-socket-dir\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623292    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-var-lib-kubelet\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-registration-dir\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623325    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-var-lib-openvswitch\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22866d7a-8cbe-4c71-8ffb-507f5d4b879f-serviceca\") pod \"node-ca-t69hm\" (UID: \"22866d7a-8cbe-4c71-8ffb-507f5d4b879f\") " pod="openshift-image-registry/node-ca-t69hm"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623380    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-sysctl-d\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623398    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-registration-dir\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623401    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55a0c1ed-dbff-455b-b25e-2fc6741f0876-iptables-alerter-script\") pod \"iptables-alerter-jmm68\" (UID: \"55a0c1ed-dbff-455b-b25e-2fc6741f0876\") " pod="openshift-network-operator/iptables-alerter-jmm68"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623458    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-var-lib-openvswitch\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623478    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/821095f1-8366-4898-9b00-1656eb70c735-ovnkube-config\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4r22\" (UniqueName: \"kubernetes.io/projected/62165010-2435-476c-bf99-75bf927b2b45-kube-api-access-x4r22\") pod \"network-metrics-daemon-7ggnb\" (UID: \"62165010-2435-476c-bf99-75bf927b2b45\") " pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:02.624827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623514    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-socket-dir\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c532b0b-be73-4f6f-be91-7ae296f21534-tmp\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623544    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-run-systemd\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r248r\" (UniqueName: \"kubernetes.io/projected/2c532b0b-be73-4f6f-be91-7ae296f21534-kube-api-access-r248r\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623587    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpmm5\" (UniqueName: \"kubernetes.io/projected/22866d7a-8cbe-4c71-8ffb-507f5d4b879f-kube-api-access-kpmm5\") pod \"node-ca-t69hm\" (UID: \"22866d7a-8cbe-4c71-8ffb-507f5d4b879f\") " pod="openshift-image-registry/node-ca-t69hm"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-sysctl-conf\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623628    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-sysctl-d\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-sys-fs\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623758    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-sysctl-conf\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623780    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-sysconfig\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30-konnectivity-ca\") pod \"konnectivity-agent-9ggjx\" (UID: \"b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30\") " pod="kube-system/konnectivity-agent-9ggjx"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623838    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g5rg\" (UniqueName: \"kubernetes.io/projected/55a0c1ed-dbff-455b-b25e-2fc6741f0876-kube-api-access-5g5rg\") pod \"iptables-alerter-jmm68\" (UID: \"55a0c1ed-dbff-455b-b25e-2fc6741f0876\") " pod="openshift-network-operator/iptables-alerter-jmm68"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-etc-openvswitch\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623870    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22866d7a-8cbe-4c71-8ffb-507f5d4b879f-serviceca\") pod \"node-ca-t69hm\" (UID: \"22866d7a-8cbe-4c71-8ffb-507f5d4b879f\") " pod="openshift-image-registry/node-ca-t69hm"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623877    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-log-socket\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.625843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623934    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-sys-fs\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-cni-bin\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623951    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-log-socket\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.623980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs\") pod \"network-metrics-daemon-7ggnb\" (UID: \"62165010-2435-476c-bf99-75bf927b2b45\") " pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.624007    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/821095f1-8366-4898-9b00-1656eb70c735-env-overrides\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.624035    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-etc-openvswitch\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.624044    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/821095f1-8366-4898-9b00-1656eb70c735-ovnkube-script-lib\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.624159    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-sysconfig\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:02.624261    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:02.624328    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs podName:62165010-2435-476c-bf99-75bf927b2b45 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:03.12430923 +0000 UTC m=+3.100140912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs") pod "network-metrics-daemon-7ggnb" (UID: "62165010-2435-476c-bf99-75bf927b2b45") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.624470    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/821095f1-8366-4898-9b00-1656eb70c735-env-overrides\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.624007    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/821095f1-8366-4898-9b00-1656eb70c735-host-cni-bin\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.624574    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/897dc98e-7462-4c2c-8579-64a0e783d074-etc-selinux\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.624599    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30-konnectivity-ca\") pod \"konnectivity-agent-9ggjx\" (UID: \"b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30\") " pod="kube-system/konnectivity-agent-9ggjx"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.624938    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/821095f1-8366-4898-9b00-1656eb70c735-ovnkube-script-lib\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.625429    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30-agent-certs\") pod \"konnectivity-agent-9ggjx\" (UID: \"b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30\") " pod="kube-system/konnectivity-agent-9ggjx"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.625834    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/821095f1-8366-4898-9b00-1656eb70c735-ovn-node-metrics-cert\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.626381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.626016    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c532b0b-be73-4f6f-be91-7ae296f21534-tmp\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.626880 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.626325    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2c532b0b-be73-4f6f-be91-7ae296f21534-etc-tuned\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.631328 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:02.631307    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:02.631328 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:02.631331    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:02.631541 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:02.631346    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zmsxm for pod openshift-network-diagnostics/network-check-target-q5kpw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:02.631541 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:02.631423    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm podName:110d78d6-4775-4717-84e3-030a8c109a9b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:03.13140406 +0000 UTC m=+3.107235759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zmsxm" (UniqueName: "kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm") pod "network-check-target-q5kpw" (UID: "110d78d6-4775-4717-84e3-030a8c109a9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:02.633066 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.632979    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95nwn\" (UniqueName: \"kubernetes.io/projected/924363b2-77bd-4de5-9279-8fdf18129796-kube-api-access-95nwn\") pod \"node-resolver-vs6m9\" (UID: \"924363b2-77bd-4de5-9279-8fdf18129796\") " pod="openshift-dns/node-resolver-vs6m9"
Aug 04 18:00:02.634865 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.634263    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g5rg\" (UniqueName: \"kubernetes.io/projected/55a0c1ed-dbff-455b-b25e-2fc6741f0876-kube-api-access-5g5rg\") pod \"iptables-alerter-jmm68\" (UID: \"55a0c1ed-dbff-455b-b25e-2fc6741f0876\") " pod="openshift-network-operator/iptables-alerter-jmm68"
Aug 04 18:00:02.634865 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.634777    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvrqc\" (UniqueName: \"kubernetes.io/projected/821095f1-8366-4898-9b00-1656eb70c735-kube-api-access-xvrqc\") pod \"ovnkube-node-8rlhj\" (UID: \"821095f1-8366-4898-9b00-1656eb70c735\") " pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.634865 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.634819    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ttb\" (UniqueName: \"kubernetes.io/projected/897dc98e-7462-4c2c-8579-64a0e783d074-kube-api-access-p9ttb\") pod \"aws-ebs-csi-driver-node-kfwk2\" (UID: \"897dc98e-7462-4c2c-8579-64a0e783d074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.635247 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.635229    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r248r\" (UniqueName: \"kubernetes.io/projected/2c532b0b-be73-4f6f-be91-7ae296f21534-kube-api-access-r248r\") pod \"tuned-7x95l\" (UID: \"2c532b0b-be73-4f6f-be91-7ae296f21534\") " pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.635294 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.635276    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpmm5\" (UniqueName: \"kubernetes.io/projected/22866d7a-8cbe-4c71-8ffb-507f5d4b879f-kube-api-access-kpmm5\") pod \"node-ca-t69hm\" (UID: \"22866d7a-8cbe-4c71-8ffb-507f5d4b879f\") " pod="openshift-image-registry/node-ca-t69hm"
Aug 04 18:00:02.635578 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.635559    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4r22\" (UniqueName: \"kubernetes.io/projected/62165010-2435-476c-bf99-75bf927b2b45-kube-api-access-x4r22\") pod \"network-metrics-daemon-7ggnb\" (UID: \"62165010-2435-476c-bf99-75bf927b2b45\") " pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:02.741216 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.741106    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5q6q7"
Aug 04 18:00:02.748071 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.748049    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zh6dz"
Aug 04 18:00:02.759329 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.759295    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:02.778003 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.777981    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jmm68"
Aug 04 18:00:02.789532 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.789513    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2"
Aug 04 18:00:02.803171 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.803153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:02.814932 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.814915    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t69hm"
Aug 04 18:00:02.828402 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.828382    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7x95l"
Aug 04 18:00:02.846367 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.846335    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vs6m9"
Aug 04 18:00:02.851620 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:02.851604    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9ggjx"
Aug 04 18:00:03.028826 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:03.028736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:03.028990 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:03.028864    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:03.028990 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:03.028928    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret podName:e9274e13-87e6-4bb8-bb29-3ed6fcde80e0 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:04.028914348 +0000 UTC m=+4.004746025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret") pod "global-pull-secret-syncer-6kdvz" (UID: "e9274e13-87e6-4bb8-bb29-3ed6fcde80e0") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:03.129327 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:03.129291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs\") pod \"network-metrics-daemon-7ggnb\" (UID: \"62165010-2435-476c-bf99-75bf927b2b45\") " pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:03.129554 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:03.129426    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:03.129554 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:03.129485    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs podName:62165010-2435-476c-bf99-75bf927b2b45 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:04.129469438 +0000 UTC m=+4.105301118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs") pod "network-metrics-daemon-7ggnb" (UID: "62165010-2435-476c-bf99-75bf927b2b45") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:03.166825 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:03.166779    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a66f6347b8909039c9c2cd29191e55.slice/crio-044301b49a675f00b2fe0f40cc63d46fd754c4fc3f7e0738b85e057818afd159 WatchSource:0}: Error finding container 044301b49a675f00b2fe0f40cc63d46fd754c4fc3f7e0738b85e057818afd159: Status 404 returned error can't find the container with id 044301b49a675f00b2fe0f40cc63d46fd754c4fc3f7e0738b85e057818afd159
Aug 04 18:00:03.170773 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:03.170755    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:00:03.229691 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:03.229644    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsxm\" (UniqueName: \"kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm\") pod \"network-check-target-q5kpw\" (UID: \"110d78d6-4775-4717-84e3-030a8c109a9b\") " pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:03.229885 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:03.229810    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:03.229885 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:03.229832    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:03.229885 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:03.229843    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zmsxm for pod openshift-network-diagnostics/network-check-target-q5kpw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:03.229995 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:03.229895    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm podName:110d78d6-4775-4717-84e3-030a8c109a9b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:04.2298818 +0000 UTC m=+4.205713477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zmsxm" (UniqueName: "kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm") pod "network-check-target-q5kpw" (UID: "110d78d6-4775-4717-84e3-030a8c109a9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:03.459833 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:03.459805    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707bcbec_504f_4e07_ae6f_6d6b4d16964e.slice/crio-048332abcd673c68493ac79f4995eb60ac6f3f9d71caa705dfdafbcca24acb54 WatchSource:0}: Error finding container 048332abcd673c68493ac79f4995eb60ac6f3f9d71caa705dfdafbcca24acb54: Status 404 returned error can't find the container with id 048332abcd673c68493ac79f4995eb60ac6f3f9d71caa705dfdafbcca24acb54
Aug 04 18:00:03.476459 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:03.476433    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821095f1_8366_4898_9b00_1656eb70c735.slice/crio-a4b5f72f4fded93bad2e7c291d1c42032498be34d072ffe71b0778c1097e1b7f WatchSource:0}: Error finding container a4b5f72f4fded93bad2e7c291d1c42032498be34d072ffe71b0778c1097e1b7f: Status 404 returned error can't find the container with id a4b5f72f4fded93bad2e7c291d1c42032498be34d072ffe71b0778c1097e1b7f
Aug 04 18:00:03.488848 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:03.488812    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:55:03 +0000 UTC" deadline="2028-02-19 07:31:53.521155097 +0000 UTC"
Aug 04 18:00:03.488934 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:03.488849    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13525h31m50.032310208s"
Aug 04 18:00:03.490096 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:03.490073    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22866d7a_8cbe_4c71_8ffb_507f5d4b879f.slice/crio-889e71df5fcd9b88fcf841729bd990fe5b27767056f5c850d3ebca393746ab05 WatchSource:0}: Error finding container 889e71df5fcd9b88fcf841729bd990fe5b27767056f5c850d3ebca393746ab05: Status 404 returned error can't find the container with id 889e71df5fcd9b88fcf841729bd990fe5b27767056f5c850d3ebca393746ab05
Aug 04 18:00:03.576229 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:03.576191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:03.576382 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:03.576325    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:03.581641 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:03.581600    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj" event={"ID":"821095f1-8366-4898-9b00-1656eb70c735","Type":"ContainerStarted","Data":"a4b5f72f4fded93bad2e7c291d1c42032498be34d072ffe71b0778c1097e1b7f"}
Aug 04 18:00:03.582784 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:03.582756    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh6dz" event={"ID":"707bcbec-504f-4e07-ae6f-6d6b4d16964e","Type":"ContainerStarted","Data":"048332abcd673c68493ac79f4995eb60ac6f3f9d71caa705dfdafbcca24acb54"}
Aug 04 18:00:03.584037 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:03.584009    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" event={"ID":"53a66f6347b8909039c9c2cd29191e55","Type":"ContainerStarted","Data":"044301b49a675f00b2fe0f40cc63d46fd754c4fc3f7e0738b85e057818afd159"}
Aug 04 18:00:03.585139 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:03.585101    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t69hm" event={"ID":"22866d7a-8cbe-4c71-8ffb-507f5d4b879f","Type":"ContainerStarted","Data":"889e71df5fcd9b88fcf841729bd990fe5b27767056f5c850d3ebca393746ab05"}
Aug 04 18:00:03.605976 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:03.605949    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897dc98e_7462_4c2c_8579_64a0e783d074.slice/crio-d3ff9135dd0095a6ba58adda33175f114e079a62e967e6a76bf534a336a834e4 WatchSource:0}: Error finding container d3ff9135dd0095a6ba58adda33175f114e079a62e967e6a76bf534a336a834e4: Status 404 returned error can't find the container with id d3ff9135dd0095a6ba58adda33175f114e079a62e967e6a76bf534a336a834e4
Aug 04 18:00:03.647773 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:03.647744    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442faadf703363fe454f012df8df1128.slice/crio-920b4560a5e0aa3e64e53de673bce5fb1ad130524fc9da1283d99ec82c1ffef2 WatchSource:0}: Error finding container 920b4560a5e0aa3e64e53de673bce5fb1ad130524fc9da1283d99ec82c1ffef2: Status 404 returned error can't find the container with id 920b4560a5e0aa3e64e53de673bce5fb1ad130524fc9da1283d99ec82c1ffef2
Aug 04 18:00:03.806573 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:03.806542    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a0c1ed_dbff_455b_b25e_2fc6741f0876.slice/crio-5eed1d6c8bfb6790528ac1e115df729ce8f16772aa01c9228985a62ce45c533f WatchSource:0}: Error finding container 5eed1d6c8bfb6790528ac1e115df729ce8f16772aa01c9228985a62ce45c533f: Status 404 returned error can't find the container with id 5eed1d6c8bfb6790528ac1e115df729ce8f16772aa01c9228985a62ce45c533f
Aug 04 18:00:04.037986 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:04.037940    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:04.038196 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:04.038128    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:04.038251 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:04.038201    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret podName:e9274e13-87e6-4bb8-bb29-3ed6fcde80e0 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:06.038181629 +0000 UTC m=+6.014013307 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret") pod "global-pull-secret-syncer-6kdvz" (UID: "e9274e13-87e6-4bb8-bb29-3ed6fcde80e0") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:04.107107 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:04.107068    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d65e6e_91ca_476e_b30e_3da05d33616e.slice/crio-7864100472bc40f332a1c4ada9a573a20786b1f11dc2c899b5b830733deff047 WatchSource:0}: Error finding container 7864100472bc40f332a1c4ada9a573a20786b1f11dc2c899b5b830733deff047: Status 404 returned error can't find the container with id 7864100472bc40f332a1c4ada9a573a20786b1f11dc2c899b5b830733deff047
Aug 04 18:00:04.138832 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:04.138796    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs\") pod \"network-metrics-daemon-7ggnb\" (UID: \"62165010-2435-476c-bf99-75bf927b2b45\") " pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:04.138990 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:04.138957    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:04.139037 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:04.139026    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs podName:62165010-2435-476c-bf99-75bf927b2b45 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:06.139009285 +0000 UTC m=+6.114840961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs") pod "network-metrics-daemon-7ggnb" (UID: "62165010-2435-476c-bf99-75bf927b2b45") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:04.233410 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:04.233381    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c40a2d_3dd2_4a5b_95f4_e8621ca93c30.slice/crio-35019db4009cc1884f1e64f90716bc6de212996fbc61c55993d152c0cfaf24dd WatchSource:0}: Error finding container 35019db4009cc1884f1e64f90716bc6de212996fbc61c55993d152c0cfaf24dd: Status 404 returned error can't find the container with id 35019db4009cc1884f1e64f90716bc6de212996fbc61c55993d152c0cfaf24dd
Aug 04 18:00:04.239695 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:04.239672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsxm\" (UniqueName: \"kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm\") pod \"network-check-target-q5kpw\" (UID: \"110d78d6-4775-4717-84e3-030a8c109a9b\") " pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:04.239841 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:04.239828    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:04.239887 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:04.239846    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:04.239887 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:04.239857    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zmsxm for pod openshift-network-diagnostics/network-check-target-q5kpw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:04.239960 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:04.239908    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm podName:110d78d6-4775-4717-84e3-030a8c109a9b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:06.239895681 +0000 UTC m=+6.215727358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zmsxm" (UniqueName: "kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm") pod "network-check-target-q5kpw" (UID: "110d78d6-4775-4717-84e3-030a8c109a9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:04.311140 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:04.311106    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c532b0b_be73_4f6f_be91_7ae296f21534.slice/crio-23e2ce45f1e79e98b32c23daa0f026a0a3c50ba71495d42d52b304f5aa238985 WatchSource:0}: Error finding container 23e2ce45f1e79e98b32c23daa0f026a0a3c50ba71495d42d52b304f5aa238985: Status 404 returned error can't find the container with id 23e2ce45f1e79e98b32c23daa0f026a0a3c50ba71495d42d52b304f5aa238985
Aug 04 18:00:04.367063 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:04.367031    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924363b2_77bd_4de5_9279_8fdf18129796.slice/crio-bdede3fe9afad62fd5cf626ecca71bc62625e67c9b4f4fee18ac706ceb9e5e70 WatchSource:0}: Error finding container bdede3fe9afad62fd5cf626ecca71bc62625e67c9b4f4fee18ac706ceb9e5e70: Status 404 returned error can't find the container with id bdede3fe9afad62fd5cf626ecca71bc62625e67c9b4f4fee18ac706ceb9e5e70
Aug 04 18:00:04.489261 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:04.489181    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:55:03 +0000 UTC" deadline="2028-02-22 11:11:32.001690969 +0000 UTC"
Aug 04 18:00:04.489261 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:04.489255    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13601h11m27.512439465s"
Aug 04 18:00:04.580258 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:04.580122    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:04.580461 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:04.580262    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:04.580461 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:04.580349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:04.580461 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:04.580449    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:04.588933 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:04.588901    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2" event={"ID":"897dc98e-7462-4c2c-8579-64a0e783d074","Type":"ContainerStarted","Data":"d3ff9135dd0095a6ba58adda33175f114e079a62e967e6a76bf534a336a834e4"}
Aug 04 18:00:04.590475 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:04.590446    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vs6m9" event={"ID":"924363b2-77bd-4de5-9279-8fdf18129796","Type":"ContainerStarted","Data":"bdede3fe9afad62fd5cf626ecca71bc62625e67c9b4f4fee18ac706ceb9e5e70"}
Aug 04 18:00:04.591992 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:04.591964    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7x95l" event={"ID":"2c532b0b-be73-4f6f-be91-7ae296f21534","Type":"ContainerStarted","Data":"23e2ce45f1e79e98b32c23daa0f026a0a3c50ba71495d42d52b304f5aa238985"}
Aug 04 18:00:04.593262 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:04.593227    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9ggjx" event={"ID":"b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30","Type":"ContainerStarted","Data":"35019db4009cc1884f1e64f90716bc6de212996fbc61c55993d152c0cfaf24dd"}
Aug 04 18:00:04.594717 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:04.594693    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5q6q7" event={"ID":"f0d65e6e-91ca-476e-b30e-3da05d33616e","Type":"ContainerStarted","Data":"7864100472bc40f332a1c4ada9a573a20786b1f11dc2c899b5b830733deff047"}
Aug 04 18:00:04.596374 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:04.596351    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jmm68" event={"ID":"55a0c1ed-dbff-455b-b25e-2fc6741f0876","Type":"ContainerStarted","Data":"5eed1d6c8bfb6790528ac1e115df729ce8f16772aa01c9228985a62ce45c533f"}
Aug 04 18:00:04.598051 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:04.598026    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal" event={"ID":"442faadf703363fe454f012df8df1128","Type":"ContainerStarted","Data":"920b4560a5e0aa3e64e53de673bce5fb1ad130524fc9da1283d99ec82c1ffef2"}
Aug 04 18:00:05.576606 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:05.576013    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:05.576606 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:05.576188    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:06.055406 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:06.055339    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:06.055612 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:06.055533    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:06.055612 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:06.055597    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret podName:e9274e13-87e6-4bb8-bb29-3ed6fcde80e0 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:10.055577932 +0000 UTC m=+10.031409612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret") pod "global-pull-secret-syncer-6kdvz" (UID: "e9274e13-87e6-4bb8-bb29-3ed6fcde80e0") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:06.156886 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:06.156297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs\") pod \"network-metrics-daemon-7ggnb\" (UID: \"62165010-2435-476c-bf99-75bf927b2b45\") " pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:06.156886 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:06.156464    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:06.156886 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:06.156530    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs podName:62165010-2435-476c-bf99-75bf927b2b45 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:10.156509925 +0000 UTC m=+10.132341605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs") pod "network-metrics-daemon-7ggnb" (UID: "62165010-2435-476c-bf99-75bf927b2b45") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:06.258432 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:06.257691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsxm\" (UniqueName: \"kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm\") pod \"network-check-target-q5kpw\" (UID: \"110d78d6-4775-4717-84e3-030a8c109a9b\") " pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:06.258432 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:06.257861    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:06.258432 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:06.257882    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:06.258432 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:06.257895    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zmsxm for pod openshift-network-diagnostics/network-check-target-q5kpw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:06.258432 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:06.257962    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm podName:110d78d6-4775-4717-84e3-030a8c109a9b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:10.257941738 +0000 UTC m=+10.233773439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zmsxm" (UniqueName: "kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm") pod "network-check-target-q5kpw" (UID: "110d78d6-4775-4717-84e3-030a8c109a9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:06.576592 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:06.576560    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:06.576791 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:06.576701    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:06.577231 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:06.577106    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:06.577302 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:06.577229    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:07.576327 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:07.576287    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:07.576537 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:07.576440    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:08.580144 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:08.579512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:08.580144 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:08.579636    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:08.580144 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:08.579709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:08.580144 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:08.579789    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:09.576648 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:09.576053    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:09.576648 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:09.576225    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:10.092916 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:10.092875    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:10.093430 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:10.093029    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:10.093430 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:10.093105    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret podName:e9274e13-87e6-4bb8-bb29-3ed6fcde80e0 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:18.0930844 +0000 UTC m=+18.068916100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret") pod "global-pull-secret-syncer-6kdvz" (UID: "e9274e13-87e6-4bb8-bb29-3ed6fcde80e0") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:10.193735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:10.193696    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs\") pod \"network-metrics-daemon-7ggnb\" (UID: \"62165010-2435-476c-bf99-75bf927b2b45\") " pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:10.193933 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:10.193873    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:10.194000 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:10.193943    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs podName:62165010-2435-476c-bf99-75bf927b2b45 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:18.193921449 +0000 UTC m=+18.169753152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs") pod "network-metrics-daemon-7ggnb" (UID: "62165010-2435-476c-bf99-75bf927b2b45") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:10.294449 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:10.294407    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsxm\" (UniqueName: \"kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm\") pod \"network-check-target-q5kpw\" (UID: \"110d78d6-4775-4717-84e3-030a8c109a9b\") " pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:10.294656 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:10.294637    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:10.294744 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:10.294664    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:10.294744 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:10.294680    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zmsxm for pod openshift-network-diagnostics/network-check-target-q5kpw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:10.294744 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:10.294743    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm podName:110d78d6-4775-4717-84e3-030a8c109a9b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:18.294723021 +0000 UTC m=+18.270554702 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zmsxm" (UniqueName: "kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm") pod "network-check-target-q5kpw" (UID: "110d78d6-4775-4717-84e3-030a8c109a9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:10.577852 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:10.577818    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:10.578045 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:10.577956    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:10.578496 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:10.578474    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:10.578602 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:10.578581    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:11.576504 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:11.576469    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:11.577029 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:11.576638    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:12.576896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:12.576861    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:12.577418 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:12.577000    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:12.577494 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:12.577448    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:12.577551 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:12.577536    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:13.576331 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:13.576040    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:13.576331 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:13.576201    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:14.576566 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:14.576525    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:14.576999 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:14.576661    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:14.576999 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:14.576718    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:14.576999 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:14.576845    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:15.575777 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:15.575750    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:15.575951 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:15.575855    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:16.576296 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:16.576263    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:16.576296 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:16.576289    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:16.576810 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:16.576389    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:16.576810 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:16.576470    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:17.576828 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:17.576790    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:17.577254 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:17.576897    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:18.155639 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:18.155603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:18.155821 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:18.155778    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:18.155879 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:18.155860    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret podName:e9274e13-87e6-4bb8-bb29-3ed6fcde80e0 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.155838105 +0000 UTC m=+34.131669833 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret") pod "global-pull-secret-syncer-6kdvz" (UID: "e9274e13-87e6-4bb8-bb29-3ed6fcde80e0") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:18.256286 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:18.256247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs\") pod \"network-metrics-daemon-7ggnb\" (UID: \"62165010-2435-476c-bf99-75bf927b2b45\") " pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:18.256474 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:18.256423    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:18.256535 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:18.256499    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs podName:62165010-2435-476c-bf99-75bf927b2b45 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.256479918 +0000 UTC m=+34.232311593 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs") pod "network-metrics-daemon-7ggnb" (UID: "62165010-2435-476c-bf99-75bf927b2b45") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:18.357152 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:18.357098    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsxm\" (UniqueName: \"kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm\") pod \"network-check-target-q5kpw\" (UID: \"110d78d6-4775-4717-84e3-030a8c109a9b\") " pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:18.357353 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:18.357258    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:18.357353 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:18.357286    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:18.357353 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:18.357300    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zmsxm for pod openshift-network-diagnostics/network-check-target-q5kpw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:18.357505 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:18.357368    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm podName:110d78d6-4775-4717-84e3-030a8c109a9b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.357349818 +0000 UTC m=+34.333181494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zmsxm" (UniqueName: "kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm") pod "network-check-target-q5kpw" (UID: "110d78d6-4775-4717-84e3-030a8c109a9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:18.576419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:18.576383    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:18.576618 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:18.576524    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:18.576618 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:18.576383    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:18.576736 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:18.576654    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:19.576681 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:19.576642    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:19.577206 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:19.576770    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:20.576779 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:20.576743    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:20.577249 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:20.576846    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:20.577249 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:20.576910    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:20.577249 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:20.577028    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:21.576620 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:21.576578    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:21.576818 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:21.576719    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:22.576693 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:22.576653    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:22.576876 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:22.576703    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:22.576876 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:22.576797    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:22.577257 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:22.576919    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:23.576480 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:23.576450    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:23.576744 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:23.576579    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:24.576277 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:24.576095    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:24.576715 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:24.576095    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:24.576715 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:24.576372    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:24.576715 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:24.576419    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:24.642761 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:24.642731    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj" event={"ID":"821095f1-8366-4898-9b00-1656eb70c735","Type":"ContainerStarted","Data":"0f03b9b70225e32635101809b2db619cded099e9b70859b6803d46545bcfbd87"}
Aug 04 18:00:24.642866 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:24.642813    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj" event={"ID":"821095f1-8366-4898-9b00-1656eb70c735","Type":"ContainerStarted","Data":"760ddd2d51dddff324844db32516d1ac849edd691c0c637cc3e5e0ca25c828dd"}
Aug 04 18:00:24.642866 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:24.642859    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj" event={"ID":"821095f1-8366-4898-9b00-1656eb70c735","Type":"ContainerStarted","Data":"caa1152b0193ef5a100f8cadd908491cf13f772eb1f2c48f17761f16b8a887e0"}
Aug 04 18:00:24.642972 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:24.642889    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj" event={"ID":"821095f1-8366-4898-9b00-1656eb70c735","Type":"ContainerStarted","Data":"30a385bff0508a0caa985d2fe2fc27262e9bad5713e9a5c9c4a7f800d3274148"}
Aug 04 18:00:24.644629 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:24.644600    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7x95l" event={"ID":"2c532b0b-be73-4f6f-be91-7ae296f21534","Type":"ContainerStarted","Data":"360ccd852005d6753470fdd6475a411f3b9b50f912b3d5bbe68b210c869e7ba9"}
Aug 04 18:00:24.650127 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:24.650089    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5q6q7" event={"ID":"f0d65e6e-91ca-476e-b30e-3da05d33616e","Type":"ContainerStarted","Data":"df98fe8fd55a524ef44d462920bd09c1f69242a5805cdc8ad957f6aa23897c56"}
Aug 04 18:00:24.652062 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:24.652037    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal" event={"ID":"442faadf703363fe454f012df8df1128","Type":"ContainerStarted","Data":"70c2d081aac3f1dd257d862272134a7d6556c6a9802ef2e9645bf3a56dd72100"}
Aug 04 18:00:24.661869 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:24.661802    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7x95l" podStartSLOduration=3.779393077 podStartE2EDuration="23.661790325s" totalImagesPullingTime="19.882397248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:24.661781155 +0000 UTC m=+24.637612851" watchObservedRunningTime="2026-08-04 18:00:24.661790325 +0000 UTC m=+24.637622021"
Aug 04 18:00:24.678756 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:24.678716    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5q6q7" podStartSLOduration=3.532655321 podStartE2EDuration="23.678701287s" totalImagesPullingTime="20.146045966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:24.678015535 +0000 UTC m=+24.653847257" watchObservedRunningTime="2026-08-04 18:00:24.678701287 +0000 UTC m=+24.654532984"
Aug 04 18:00:25.576289 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.576257    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:25.576865 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:25.576375    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:25.655823 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.655793    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9ggjx" event={"ID":"b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30","Type":"ContainerStarted","Data":"5e51e5246af37019ee057ebbed54d0eee5014a3ae043281731cfcd84e719455e"}
Aug 04 18:00:25.657261 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.657239    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t69hm" event={"ID":"22866d7a-8cbe-4c71-8ffb-507f5d4b879f","Type":"ContainerStarted","Data":"e6d92fe348029dd9957aa5a481012cb6af1b84f06c598cd8d86eb9f2abb19bc8"}
Aug 04 18:00:25.658518 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.658495    2576 generic.go:356] "Generic (PLEG): container finished" podID="53a66f6347b8909039c9c2cd29191e55" containerID="4cdd68a0bf6a9f7b52be6d29d473e9300b90729b02c284948b9ba34a801105f3" exitCode=0
Aug 04 18:00:25.658609 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.658561    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" event={"ID":"53a66f6347b8909039c9c2cd29191e55","Type":"ContainerDied","Data":"4cdd68a0bf6a9f7b52be6d29d473e9300b90729b02c284948b9ba34a801105f3"}
Aug 04 18:00:25.659875 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.659853    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2" event={"ID":"897dc98e-7462-4c2c-8579-64a0e783d074","Type":"ContainerStarted","Data":"cdc4d1503cfbf8aef0e4f9481a9a4dd3a802fa3ca3f42a27fa8701e8dde1d348"}
Aug 04 18:00:25.662870 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.662852    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj" event={"ID":"821095f1-8366-4898-9b00-1656eb70c735","Type":"ContainerStarted","Data":"7c01575dd8301c1a75b87c4d0eba4e841272b873e27860b4147452bc3ed8d9e4"}
Aug 04 18:00:25.662964 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.662904    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj" event={"ID":"821095f1-8366-4898-9b00-1656eb70c735","Type":"ContainerStarted","Data":"e966854c54f0dd411e7325a32d2b757f4b4ee58b3dbf945531d05f1c0cc6d7c0"}
Aug 04 18:00:25.664224 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.664204    2576 generic.go:356] "Generic (PLEG): container finished" podID="707bcbec-504f-4e07-ae6f-6d6b4d16964e" containerID="69bd7970d369452a4cbedcef65ba3795990498c0d8e2bd5cdba88401cbbb8262" exitCode=0
Aug 04 18:00:25.664303 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.664273    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh6dz" event={"ID":"707bcbec-504f-4e07-ae6f-6d6b4d16964e","Type":"ContainerDied","Data":"69bd7970d369452a4cbedcef65ba3795990498c0d8e2bd5cdba88401cbbb8262"}
Aug 04 18:00:25.665551 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.665527    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vs6m9" event={"ID":"924363b2-77bd-4de5-9279-8fdf18129796","Type":"ContainerStarted","Data":"75ad9e5260f892f4a38dda92a6f9a6d7410834867b652318fca6998be0ba9e2a"}
Aug 04 18:00:25.675869 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.675833    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal" podStartSLOduration=23.675822309 podStartE2EDuration="23.675822309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:24.691991005 +0000 UTC m=+24.667822703" watchObservedRunningTime="2026-08-04 18:00:25.675822309 +0000 UTC m=+25.651654006"
Aug 04 18:00:25.698031 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.697993    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9ggjx" podStartSLOduration=4.746442146 podStartE2EDuration="24.69797958s" totalImagesPullingTime="19.951537434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:25.675523257 +0000 UTC m=+25.651354945" watchObservedRunningTime="2026-08-04 18:00:25.69797958 +0000 UTC m=+25.673811276"
Aug 04 18:00:25.726801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.726753    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t69hm" podStartSLOduration=4.06169467 podStartE2EDuration="24.726733669s" totalImagesPullingTime="20.665038999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:25.726245684 +0000 UTC m=+25.702077380" watchObservedRunningTime="2026-08-04 18:00:25.726733669 +0000 UTC m=+25.702565367"
Aug 04 18:00:25.955242 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:25.955207    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:26.034125 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:26.034075    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:26.468128 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:26.468071    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:26.576543 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:26.576516    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:26.576943 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:26.576515    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:26.576943 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:26.576650    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:26.576943 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:26.576736    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:26.669732 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:26.669685    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jmm68" event={"ID":"55a0c1ed-dbff-455b-b25e-2fc6741f0876","Type":"ContainerStarted","Data":"ccb702f9c068ec33d40db3e562259c7a2e1258d5514e3e78291d3cea5f17304c"}
Aug 04 18:00:26.672091 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:26.672058    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" event={"ID":"53a66f6347b8909039c9c2cd29191e55","Type":"ContainerStarted","Data":"60dcf6647f8966840956fc7b2e56d5de883668a2543034978e1582470104c7e2"}
Aug 04 18:00:26.690586 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:26.690538    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vs6m9" podStartSLOduration=5.861334092 podStartE2EDuration="25.690524051s" totalImagesPullingTime="19.829189959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:25.742698194 +0000 UTC m=+25.718529890" watchObservedRunningTime="2026-08-04 18:00:26.690524051 +0000 UTC m=+26.666355776"
Aug 04 18:00:26.691173 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:26.691142    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jmm68" podStartSLOduration=13.03053967 podStartE2EDuration="25.69113223s" totalImagesPullingTime="12.66059256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:26.690407619 +0000 UTC m=+26.666239316" watchObservedRunningTime="2026-08-04 18:00:26.69113223 +0000 UTC m=+26.666963919"
Aug 04 18:00:26.777155 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:26.777130    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 04 18:00:27.253753 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:27.253649    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:27.325495 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:27.325437    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:27.485634 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:27.485494    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T18:00:26.777149035Z","UUID":"b23fa623-bc2f-4186-9c6d-536db20439e2","Handler":null,"Name":"","Endpoint":""}
Aug 04 18:00:27.487635 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:27.487586    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 04 18:00:27.487851 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:27.487832    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 04 18:00:27.576389 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:27.576360    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:27.576578 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:27.576503    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:27.608195 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:27.608159    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:27.677614 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:27.677578    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj" event={"ID":"821095f1-8366-4898-9b00-1656eb70c735","Type":"ContainerStarted","Data":"2d2a28141df3a8c739fbe33bd2b47c30bb02d6bb052c3dfb219316027aea7454"}
Aug 04 18:00:27.679522 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:27.679496    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2" event={"ID":"897dc98e-7462-4c2c-8579-64a0e783d074","Type":"ContainerStarted","Data":"1a4367dbe8825bc61b182f1c74a5ae12b070299f6f274581164577d809cfa8f5"}
Aug 04 18:00:28.075463 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:28.075416    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:28.178592 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:28.178552    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:28.576574 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:28.576543    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:28.577021 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:28.576672    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:28.577021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:28.576714    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:28.577021 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:28.576810    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:28.683901 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:28.683823    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2" event={"ID":"897dc98e-7462-4c2c-8579-64a0e783d074","Type":"ContainerStarted","Data":"3a5a23713bfc61e1c661bea60ec4964ab346163192fc0505866d0f49e5301ec6"}
Aug 04 18:00:28.705882 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:28.705832    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfwk2" podStartSLOduration=4.089750799 podStartE2EDuration="27.705816997s" totalImagesPullingTime="23.616066198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:28.705231958 +0000 UTC m=+28.681063678" watchObservedRunningTime="2026-08-04 18:00:28.705816997 +0000 UTC m=+28.681648694"
Aug 04 18:00:28.706076 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:28.706047    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" podStartSLOduration=26.706037324 podStartE2EDuration="26.706037324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:26.718875031 +0000 UTC m=+26.694706730" watchObservedRunningTime="2026-08-04 18:00:28.706037324 +0000 UTC m=+28.681869022"
Aug 04 18:00:29.575765 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:29.575733    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:29.576001 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:29.575874    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:29.671100 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:29.671075    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9ggjx"
Aug 04 18:00:29.671600 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:29.671584    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9ggjx"
Aug 04 18:00:29.685767 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:29.685746    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9ggjx"
Aug 04 18:00:29.685996 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:29.685972    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9ggjx"
Aug 04 18:00:30.523667 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:30.523498    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:30.577153 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:30.577101    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:30.577302 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:30.577166    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:30.577302 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:30.577233    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:30.577374 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:30.577339    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:30.691693 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:30.691655    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj" event={"ID":"821095f1-8366-4898-9b00-1656eb70c735","Type":"ContainerStarted","Data":"fa5ce7cbb512d93afec9ec0ba7bbde4ca380a754d93b22531fdf01f3c49c6828"}
Aug 04 18:00:30.692491 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:30.691978    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:30.692491 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:30.692066    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:30.693535 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:30.693498    2576 generic.go:356] "Generic (PLEG): container finished" podID="707bcbec-504f-4e07-ae6f-6d6b4d16964e" containerID="d445e0f8dd4ddd205547a29efe25d4e18cbc51d2118e043fdf9030344a2f9138" exitCode=0
Aug 04 18:00:30.693666 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:30.693571    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh6dz" event={"ID":"707bcbec-504f-4e07-ae6f-6d6b4d16964e","Type":"ContainerDied","Data":"d445e0f8dd4ddd205547a29efe25d4e18cbc51d2118e043fdf9030344a2f9138"}
Aug 04 18:00:30.707995 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:30.707976    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:30.718339 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:30.718299    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj" podStartSLOduration=8.935810661 podStartE2EDuration="29.718284445s" totalImagesPullingTime="20.782473784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:30.717093552 +0000 UTC m=+30.692925249" watchObservedRunningTime="2026-08-04 18:00:30.718284445 +0000 UTC m=+30.694116136"
Aug 04 18:00:31.575887 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:31.575727    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:31.576070 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:31.575964    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:31.609711 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:31.608356    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:31.678793 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:31.678760    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6kdvz"]
Aug 04 18:00:31.678983 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:31.678968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:31.679140 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:31.679085    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:31.679565 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:31.679532    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q5kpw"]
Aug 04 18:00:31.679703 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:31.679689    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:31.679865 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:31.679828    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:31.680498 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:31.680479    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7ggnb"]
Aug 04 18:00:31.682747 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:31.682713    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:31.698182 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:31.698081    2576 generic.go:356] "Generic (PLEG): container finished" podID="707bcbec-504f-4e07-ae6f-6d6b4d16964e" containerID="ba026ff1220bb3a4a7c2c68df5e9c5f2f06243568375297c7135f5186a59efba" exitCode=0
Aug 04 18:00:31.698182 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:31.698142    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh6dz" event={"ID":"707bcbec-504f-4e07-ae6f-6d6b4d16964e","Type":"ContainerDied","Data":"ba026ff1220bb3a4a7c2c68df5e9c5f2f06243568375297c7135f5186a59efba"}
Aug 04 18:00:31.698683 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:31.698252    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:31.698683 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:31.698587    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:31.699437 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:31.699140    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:31.715984 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:31.715959    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:00:33.144565 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:33.144533    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:33.309003 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:33.308968    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:33.376281 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:33.376243    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:33.467453 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:33.467415    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:33.575976 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:33.575905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:33.575976 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:33.575963    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:33.576151 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:33.575918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:33.576151 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:33.576022    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:33.576151 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:33.576074    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:33.576248 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:33.576174    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:33.704592 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:33.704562    2576 generic.go:356] "Generic (PLEG): container finished" podID="707bcbec-504f-4e07-ae6f-6d6b4d16964e" containerID="1a76a6afa70bb816adfc37688cb85dd0af3657123879d87f9eb51905449f0154" exitCode=0
Aug 04 18:00:33.704780 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:33.704645    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh6dz" event={"ID":"707bcbec-504f-4e07-ae6f-6d6b4d16964e","Type":"ContainerDied","Data":"1a76a6afa70bb816adfc37688cb85dd0af3657123879d87f9eb51905449f0154"}
Aug 04 18:00:34.170099 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:34.169996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:34.170502 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:34.170174    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:34.170502 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:34.170240    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret podName:e9274e13-87e6-4bb8-bb29-3ed6fcde80e0 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:06.170224425 +0000 UTC m=+66.146056099 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret") pod "global-pull-secret-syncer-6kdvz" (UID: "e9274e13-87e6-4bb8-bb29-3ed6fcde80e0") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:34.270372 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:34.270333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs\") pod \"network-metrics-daemon-7ggnb\" (UID: \"62165010-2435-476c-bf99-75bf927b2b45\") " pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:34.270543 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:34.270449    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:34.270543 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:34.270503    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs podName:62165010-2435-476c-bf99-75bf927b2b45 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:06.270488179 +0000 UTC m=+66.246319854 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs") pod "network-metrics-daemon-7ggnb" (UID: "62165010-2435-476c-bf99-75bf927b2b45") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:34.371314 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:34.371279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsxm\" (UniqueName: \"kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm\") pod \"network-check-target-q5kpw\" (UID: \"110d78d6-4775-4717-84e3-030a8c109a9b\") " pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:34.371451 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:34.371431    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:34.371451 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:34.371451    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:34.371534 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:34.371460    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zmsxm for pod openshift-network-diagnostics/network-check-target-q5kpw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:34.371534 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:34.371512    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm podName:110d78d6-4775-4717-84e3-030a8c109a9b nodeName:}" failed. No retries permitted until 2026-08-04 18:01:06.371495642 +0000 UTC m=+66.347327332 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zmsxm" (UniqueName: "kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm") pod "network-check-target-q5kpw" (UID: "110d78d6-4775-4717-84e3-030a8c109a9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:34.630816 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:34.630785    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:34.730677 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:34.730626    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:34.802211 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:34.802183    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vs6m9_924363b2-77bd-4de5-9279-8fdf18129796/dns-node-resolver/0.log"
Aug 04 18:00:35.100313 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:35.100281    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:35.167350 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:35.167317    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:35.576498 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:35.576467    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:35.576931 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:35.576467    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:35.576931 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:35.576610    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:35.576931 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:35.576467    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:35.576931 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:35.576704    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:35.576931 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:35.576740    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:35.802162 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:35.802135    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t69hm_22866d7a-8cbe-4c71-8ffb-507f5d4b879f/node-ca/0.log"
Aug 04 18:00:36.003135 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:36.003025    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:37.576523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:37.576485    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:37.577050 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:37.576485    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:37.577050 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:37.576619    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:37.577050 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:37.576485    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:37.577050 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:37.576686    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:37.577050 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:37.576770    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:37.900231 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:37.900146    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:38.718728 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:38.718691    2576 generic.go:356] "Generic (PLEG): container finished" podID="707bcbec-504f-4e07-ae6f-6d6b4d16964e" containerID="fdc7f5b4ec2d29e37ffc9924dbc071ee9ca51d983af9f5d469e6cdcea3d17d66" exitCode=0
Aug 04 18:00:38.719103 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:38.718739    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh6dz" event={"ID":"707bcbec-504f-4e07-ae6f-6d6b4d16964e","Type":"ContainerDied","Data":"fdc7f5b4ec2d29e37ffc9924dbc071ee9ca51d983af9f5d469e6cdcea3d17d66"}
Aug 04 18:00:38.911528 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:38.911344    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:39.048168 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:39.048125    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:39.518670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:39.518587    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:39.576247 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:39.576212    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:39.576420 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:39.576229    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:39.576420 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:39.576331    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:39.576420 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:39.576327    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:39.576420 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:39.576400    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:39.576606 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:39.576523    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:39.653999 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:39.653962    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:39.723383 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:39.723348    2576 generic.go:356] "Generic (PLEG): container finished" podID="707bcbec-504f-4e07-ae6f-6d6b4d16964e" containerID="a9cd4ff70368f87c99908c3f40694eee20b7158cdb6846075aaeadce6e321a0b" exitCode=0
Aug 04 18:00:39.723732 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:39.723424    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh6dz" event={"ID":"707bcbec-504f-4e07-ae6f-6d6b4d16964e","Type":"ContainerDied","Data":"a9cd4ff70368f87c99908c3f40694eee20b7158cdb6846075aaeadce6e321a0b"}
Aug 04 18:00:39.780681 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:39.780647    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:40.728954 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:40.728922    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh6dz" event={"ID":"707bcbec-504f-4e07-ae6f-6d6b4d16964e","Type":"ContainerStarted","Data":"5fdd2828688bc868c939a2531368305d9e2ba6c824865d821dfc02a9799c4bbd"}
Aug 04 18:00:40.763290 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:40.763222    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zh6dz" podStartSLOduration=8.271458006 podStartE2EDuration="39.763206619s" totalImagesPullingTime="31.491748613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:40.762819291 +0000 UTC m=+40.738650988" watchObservedRunningTime="2026-08-04 18:00:40.763206619 +0000 UTC m=+40.739038316"
Aug 04 18:00:40.820386 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:40.820352    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:41.009443 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:41.009358    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:41.201906 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:41.201831    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:41.384414 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:41.384383    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:41.576580 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:41.576549    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:41.576767 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:41.576649    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:41.576767 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:41.576662    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:41.576767 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:41.576673    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:41.576767 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:41.576733    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:41.576978 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:41.576809    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:41.608041 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:41.608012    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:42.159744 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:42.159713    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:42.224453 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:42.224421    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:42.309440 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:42.309402    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:42.609302 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:42.609270    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:42.797691 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:42.797654    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:42.997075 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:42.996998    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:43.189253 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:43.189202    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:43.309252 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:43.309225    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:43.394441 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:43.394407    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:43.576792 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:43.576709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:43.576792 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:43.576727    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:43.576988 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:43.576709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:43.576988 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:43.576824    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:43.576988 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:43.576884    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:43.576988 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:43.576945    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:43.584606 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:43.584576    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:44.378521 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:44.378487    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:44.785809 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:44.785720    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:45.183927 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:45.183895    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:45.285543 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:45.285500    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:45.576255 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:45.576228    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:45.576711 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:45.576329    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:45.576711 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:45.576350    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:45.576711 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:45.576331    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:45.576711 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:45.576443    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:45.576711 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:45.576532    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:46.640938 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:46.640905    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:46.705530 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:46.705495    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:46.985559 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:46.985473    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:47.208739 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:47.208702    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:47.576026 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:47.575991    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:47.576026 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:47.576031    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:47.576252 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:47.576102    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:47.576252 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:47.576140    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:47.576252 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:47.576214    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:47.576346 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:47.576268    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:47.585187 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:47.585164    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:47.786745 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:47.786715    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:47.905155 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:47.905059    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:48.004911 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:48.004878    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:48.085284 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:48.085250    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:48.470207 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:48.470177    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:49.000511 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:49.000478    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:49.195910 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:49.195873    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:49.385740 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:49.385683    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:49.576484 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:49.576451    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:49.576662 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:49.576550    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:49.576662 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:49.576558    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:49.576662 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:49.576579    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:49.576789 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:49.576661    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:49.576789 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:49.576736    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:49.582952 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:49.582920    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:51.575823 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:51.575781    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:51.575823 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:51.575820    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:51.576280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:51.575782    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:51.576280 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:51.575891    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:51.576280 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:51.576003    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:51.576280 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:51.576064    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:53.576371 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:53.576340    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:53.576811 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:53.576430    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:53.576811 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:53.576436    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:53.576811 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:53.576455    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:53.576811 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:53.576515    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:53.576811 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:53.576593    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:54.403564 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:54.403531    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:54.505999 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:54.505955    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:54.653374 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:54.653338    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:55.035639 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:55.035606    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:55.135457 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:55.135416    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:55.576637 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:55.576607    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:55.576809 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:55.576707    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:55.576809 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:55.576717    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:55.576809 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:55.576736    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:55.576809 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:55.576798    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:55.576942 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:55.576854    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:56.094692 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:56.094657    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:56.156197 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:56.156167    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:56.281245 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:56.281210    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:56.482082 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:56.482006    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:56.538892 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:56.538859    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:57.105726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.105670    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:57.576624 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.576585    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:57.576788 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.576730    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:57.576788 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:57.576735    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5kpw" podUID="110d78d6-4775-4717-84e3-030a8c109a9b"
Aug 04 18:00:57.576788 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.576758    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:57.576887 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:57.576844    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6kdvz" podUID="e9274e13-87e6-4bb8-bb29-3ed6fcde80e0"
Aug 04 18:00:57.576969 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:00:57.576918    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7ggnb" podUID="62165010-2435-476c-bf99-75bf927b2b45"
Aug 04 18:00:57.811847 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.811826    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeReady"
Aug 04 18:00:57.811957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.811948    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 04 18:00:57.857738 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.857674    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w7w5n"]
Aug 04 18:00:57.859312 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.859298    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:00:57.860574 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.860549    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s4m69"]
Aug 04 18:00:57.862137 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.862104    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-snn7x\""
Aug 04 18:00:57.862244 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.862169    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 04 18:00:57.862380 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.862362    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 04 18:00:57.862441 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.862428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:57.865811 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.865796    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 04 18:00:57.865919 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.865901    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 04 18:00:57.865964 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.865909    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sm4mb\""
Aug 04 18:00:57.866619 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.866604    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 04 18:00:57.866787 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.866774    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 04 18:00:57.870335 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.870312    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w7w5n"]
Aug 04 18:00:57.874236 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.873861    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4m69"]
Aug 04 18:00:57.963518 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.963488    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-27kng"]
Aug 04 18:00:57.965159 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.965145    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-27kng"
Aug 04 18:00:57.967777 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.967755    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 04 18:00:57.967906 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.967888    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 04 18:00:57.967973 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.967926    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 04 18:00:57.968036 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.967980    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lcczt\""
Aug 04 18:00:57.975729 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:57.975708    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-27kng"]
Aug 04 18:00:58.041658 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.041606    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a-metrics-tls\") pod \"dns-default-w7w5n\" (UID: \"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a\") " pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:00:58.041842 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.041706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a-tmp-dir\") pod \"dns-default-w7w5n\" (UID: \"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a\") " pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:00:58.041842 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.041737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25b94f13-bec2-41c1-a848-37a3d18170a9-data-volume\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.041842 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.041767    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25b94f13-bec2-41c1-a848-37a3d18170a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.041842 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.041821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a-config-volume\") pod \"dns-default-w7w5n\" (UID: \"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a\") " pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:00:58.042031 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.041848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d8tk\" (UniqueName: \"kubernetes.io/projected/cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a-kube-api-access-4d8tk\") pod \"dns-default-w7w5n\" (UID: \"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a\") " pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:00:58.042031 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.041875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hctd6\" (UniqueName: \"kubernetes.io/projected/25b94f13-bec2-41c1-a848-37a3d18170a9-kube-api-access-hctd6\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.042031 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.041947    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25b94f13-bec2-41c1-a848-37a3d18170a9-crio-socket\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.042031 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.042000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25b94f13-bec2-41c1-a848-37a3d18170a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.142892 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.142818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a-config-volume\") pod \"dns-default-w7w5n\" (UID: \"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a\") " pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:00:58.142892 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.142860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3155211c-54d6-4956-9506-85213a63de71-cert\") pod \"ingress-canary-27kng\" (UID: \"3155211c-54d6-4956-9506-85213a63de71\") " pod="openshift-ingress-canary/ingress-canary-27kng"
Aug 04 18:00:58.143363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.142897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8tk\" (UniqueName: \"kubernetes.io/projected/cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a-kube-api-access-4d8tk\") pod \"dns-default-w7w5n\" (UID: \"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a\") " pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:00:58.143363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.142923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hctd6\" (UniqueName: \"kubernetes.io/projected/25b94f13-bec2-41c1-a848-37a3d18170a9-kube-api-access-hctd6\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.143363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.142943    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25b94f13-bec2-41c1-a848-37a3d18170a9-crio-socket\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.143363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.142963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25b94f13-bec2-41c1-a848-37a3d18170a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.143363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.142984    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a-metrics-tls\") pod \"dns-default-w7w5n\" (UID: \"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a\") " pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:00:58.143363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.143153    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmtt\" (UniqueName: \"kubernetes.io/projected/3155211c-54d6-4956-9506-85213a63de71-kube-api-access-chmtt\") pod \"ingress-canary-27kng\" (UID: \"3155211c-54d6-4956-9506-85213a63de71\") " pod="openshift-ingress-canary/ingress-canary-27kng"
Aug 04 18:00:58.143363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.143222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a-tmp-dir\") pod \"dns-default-w7w5n\" (UID: \"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a\") " pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:00:58.143363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.143257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25b94f13-bec2-41c1-a848-37a3d18170a9-data-volume\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.143363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.143285    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25b94f13-bec2-41c1-a848-37a3d18170a9-crio-socket\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.143363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.143288    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25b94f13-bec2-41c1-a848-37a3d18170a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.143750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.143543    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a-tmp-dir\") pod \"dns-default-w7w5n\" (UID: \"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a\") " pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:00:58.143750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.143549    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a-config-volume\") pod \"dns-default-w7w5n\" (UID: \"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a\") " pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:00:58.143750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.143589    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25b94f13-bec2-41c1-a848-37a3d18170a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.143750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.143656    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25b94f13-bec2-41c1-a848-37a3d18170a9-data-volume\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.147119 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.147088    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a-metrics-tls\") pod \"dns-default-w7w5n\" (UID: \"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a\") " pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:00:58.147172 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.147138    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25b94f13-bec2-41c1-a848-37a3d18170a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.150448 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.150429    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d8tk\" (UniqueName: \"kubernetes.io/projected/cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a-kube-api-access-4d8tk\") pod \"dns-default-w7w5n\" (UID: \"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a\") " pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:00:58.150672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.150652    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hctd6\" (UniqueName: \"kubernetes.io/projected/25b94f13-bec2-41c1-a848-37a3d18170a9-kube-api-access-hctd6\") pod \"insights-runtime-extractor-s4m69\" (UID: \"25b94f13-bec2-41c1-a848-37a3d18170a9\") " pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.170673 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.170646    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:00:58.177131 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.177098    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4m69"
Aug 04 18:00:58.188225 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.188176    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:58.244100 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.243814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3155211c-54d6-4956-9506-85213a63de71-cert\") pod \"ingress-canary-27kng\" (UID: \"3155211c-54d6-4956-9506-85213a63de71\") " pod="openshift-ingress-canary/ingress-canary-27kng"
Aug 04 18:00:58.244100 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.243905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmtt\" (UniqueName: \"kubernetes.io/projected/3155211c-54d6-4956-9506-85213a63de71-kube-api-access-chmtt\") pod \"ingress-canary-27kng\" (UID: \"3155211c-54d6-4956-9506-85213a63de71\") " pod="openshift-ingress-canary/ingress-canary-27kng"
Aug 04 18:00:58.247165 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.247092    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3155211c-54d6-4956-9506-85213a63de71-cert\") pod \"ingress-canary-27kng\" (UID: \"3155211c-54d6-4956-9506-85213a63de71\") " pod="openshift-ingress-canary/ingress-canary-27kng"
Aug 04 18:00:58.253264 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.253014    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmtt\" (UniqueName: \"kubernetes.io/projected/3155211c-54d6-4956-9506-85213a63de71-kube-api-access-chmtt\") pod \"ingress-canary-27kng\" (UID: \"3155211c-54d6-4956-9506-85213a63de71\") " pod="openshift-ingress-canary/ingress-canary-27kng"
Aug 04 18:00:58.254023 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.253991    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:58.273467 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.273440    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-27kng"
Aug 04 18:00:58.321089 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.321060    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w7w5n"]
Aug 04 18:00:58.335326 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.334351    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4m69"]
Aug 04 18:00:58.339141 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:58.339094    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b94f13_bec2_41c1_a848_37a3d18170a9.slice/crio-23a2481f376bd9d92b1acd91dc1d80142e6e023a432792401d84247c49c9747a WatchSource:0}: Error finding container 23a2481f376bd9d92b1acd91dc1d80142e6e023a432792401d84247c49c9747a: Status 404 returned error can't find the container with id 23a2481f376bd9d92b1acd91dc1d80142e6e023a432792401d84247c49c9747a
Aug 04 18:00:58.406760 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.406688    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-27kng"]
Aug 04 18:00:58.409627 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:00:58.409596    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3155211c_54d6_4956_9506_85213a63de71.slice/crio-27e1bf3413900be8a752bc53eea79572baea996b008c9f6519b0580ff75178b1 WatchSource:0}: Error finding container 27e1bf3413900be8a752bc53eea79572baea996b008c9f6519b0580ff75178b1: Status 404 returned error can't find the container with id 27e1bf3413900be8a752bc53eea79572baea996b008c9f6519b0580ff75178b1
Aug 04 18:00:58.770988 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.770912    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-27kng" event={"ID":"3155211c-54d6-4956-9506-85213a63de71","Type":"ContainerStarted","Data":"27e1bf3413900be8a752bc53eea79572baea996b008c9f6519b0580ff75178b1"}
Aug 04 18:00:58.772576 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.772547    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4m69" event={"ID":"25b94f13-bec2-41c1-a848-37a3d18170a9","Type":"ContainerStarted","Data":"233bf825967c29804a35e380549ce9dff01b239c601da57e63f2757bec2da021"}
Aug 04 18:00:58.772703 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.772600    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4m69" event={"ID":"25b94f13-bec2-41c1-a848-37a3d18170a9","Type":"ContainerStarted","Data":"23a2481f376bd9d92b1acd91dc1d80142e6e023a432792401d84247c49c9747a"}
Aug 04 18:00:58.773757 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:58.773733    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7w5n" event={"ID":"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a","Type":"ContainerStarted","Data":"340e1670e89baf0c806225e271a050d7fad2c6f4bc55ca813cb3876b35023c84"}
Aug 04 18:00:59.576105 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:59.576070    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:00:59.576105 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:59.576085    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:00:59.576602 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:59.576134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:00:59.582426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:59.582405    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:00:59.583497 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:59.583350    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qxtnm\""
Aug 04 18:00:59.583868 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:59.583605    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-88fc8\""
Aug 04 18:00:59.583868 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:59.583666    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 18:00:59.583868 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:59.583811    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:00:59.584015 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:00:59.583929    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 18:01:00.503342 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.503229    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:00.538488 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.538454    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-fh84v"]
Aug 04 18:01:00.540438 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.540417    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-fh84v"
Aug 04 18:01:00.542989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.542969    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Aug 04 18:01:00.543097 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.543026    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zznhx\""
Aug 04 18:01:00.551587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.551564    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-fh84v"]
Aug 04 18:01:00.604421 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.604384    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:00.660521 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.660486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f220e635-729a-4535-aa7a-27ffe2bc2fb1-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-fh84v\" (UID: \"f220e635-729a-4535-aa7a-27ffe2bc2fb1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-fh84v"
Aug 04 18:01:00.761728 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.761628    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f220e635-729a-4535-aa7a-27ffe2bc2fb1-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-fh84v\" (UID: \"f220e635-729a-4535-aa7a-27ffe2bc2fb1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-fh84v"
Aug 04 18:01:00.761896 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:01:00.761795    2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Aug 04 18:01:00.761896 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:01:00.761878    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f220e635-729a-4535-aa7a-27ffe2bc2fb1-tls-certificates podName:f220e635-729a-4535-aa7a-27ffe2bc2fb1 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:01.261858002 +0000 UTC m=+61.237689680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/f220e635-729a-4535-aa7a-27ffe2bc2fb1-tls-certificates") pod "prometheus-operator-admission-webhook-66685ff6d4-fh84v" (UID: "f220e635-729a-4535-aa7a-27ffe2bc2fb1") : secret "prometheus-operator-admission-webhook-tls" not found
Aug 04 18:01:00.780390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.780357    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4m69" event={"ID":"25b94f13-bec2-41c1-a848-37a3d18170a9","Type":"ContainerStarted","Data":"b8bb0c5321861a37c1de045fa14b240b0c0258be7b52cbdd608bdc886ed2637c"}
Aug 04 18:01:00.781883 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.781859    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7w5n" event={"ID":"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a","Type":"ContainerStarted","Data":"495bb3ece92bbaac8da4c7020c43b1ea584f817b910ff43aa3b47024e259e86b"}
Aug 04 18:01:00.782007 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.781898    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7w5n" event={"ID":"cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a","Type":"ContainerStarted","Data":"f941541cc5c8861fbfde333bb50b14f1d1a80401c4a0c6f3f76dfffdf6d2d2c5"}
Aug 04 18:01:00.782055 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.782003    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:01:00.798405 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.798366    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w7w5n" podStartSLOduration=2.373987095 podStartE2EDuration="3.798351223s" totalImagesPullingTime="1.424364128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:00.797851969 +0000 UTC m=+60.773683665" watchObservedRunningTime="2026-08-04 18:01:00.798351223 +0000 UTC m=+60.774182903"
Aug 04 18:01:00.892840 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:00.892797    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:01.265425 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:01.265382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f220e635-729a-4535-aa7a-27ffe2bc2fb1-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-fh84v\" (UID: \"f220e635-729a-4535-aa7a-27ffe2bc2fb1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-fh84v"
Aug 04 18:01:01.267755 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:01.267737    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f220e635-729a-4535-aa7a-27ffe2bc2fb1-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-fh84v\" (UID: \"f220e635-729a-4535-aa7a-27ffe2bc2fb1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-fh84v"
Aug 04 18:01:01.450382 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:01.450331    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-fh84v"
Aug 04 18:01:01.574260 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:01.574229    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-fh84v"]
Aug 04 18:01:01.577145 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:01.577102    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf220e635_729a_4535_aa7a_27ffe2bc2fb1.slice/crio-183b3887355385189151417188e3474ccbd3b8c6f9e82215478315d7f639b405 WatchSource:0}: Error finding container 183b3887355385189151417188e3474ccbd3b8c6f9e82215478315d7f639b405: Status 404 returned error can't find the container with id 183b3887355385189151417188e3474ccbd3b8c6f9e82215478315d7f639b405
Aug 04 18:01:01.785801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:01.785716    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-fh84v" event={"ID":"f220e635-729a-4535-aa7a-27ffe2bc2fb1","Type":"ContainerStarted","Data":"183b3887355385189151417188e3474ccbd3b8c6f9e82215478315d7f639b405"}
Aug 04 18:01:02.659482 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:02.659432    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:02.761012 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:02.760973    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:02.863890 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:02.863839    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:02.962667 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:02.962622    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:03.070594 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.070559    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:03.414866 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.414823    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:03.438872 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.438833    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dc4c4ccd9-kv9l7"]
Aug 04 18:01:03.454269 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.454233    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.454726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.454641    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dc4c4ccd9-kv9l7"]
Aug 04 18:01:03.458600 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.458482    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Aug 04 18:01:03.458600 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.458503    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Aug 04 18:01:03.458600 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.458520    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Aug 04 18:01:03.460827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.460471    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Aug 04 18:01:03.460827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.460586    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Aug 04 18:01:03.460827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.460596    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Aug 04 18:01:03.460827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.460637    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jw8gn\""
Aug 04 18:01:03.460827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.460683    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Aug 04 18:01:03.479934 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.479903    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:03.584969 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.584927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-oauth-serving-cert\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.585263 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.584996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-serving-cert\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.585263 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.585024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-service-ca\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.585263 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.585055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-config\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.585263 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.585176    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-oauth-config\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.585263 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.585253    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jwfd\" (UniqueName: \"kubernetes.io/projected/716abfd2-ee03-4ebc-bb89-d41d82d26090-kube-api-access-5jwfd\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.686019 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.685943    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-serving-cert\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.686019 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.685982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-service-ca\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.686019 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.686001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-config\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.686411 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.686172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-oauth-config\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.686411 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.686225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jwfd\" (UniqueName: \"kubernetes.io/projected/716abfd2-ee03-4ebc-bb89-d41d82d26090-kube-api-access-5jwfd\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.686411 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.686289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-oauth-serving-cert\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.686790 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.686768    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-service-ca\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.686849 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.686806    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-config\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.686909 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.686893    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-oauth-serving-cert\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.689005 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.688979    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-oauth-config\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.689100 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.689036    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-serving-cert\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.695513 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.695492    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jwfd\" (UniqueName: \"kubernetes.io/projected/716abfd2-ee03-4ebc-bb89-d41d82d26090-kube-api-access-5jwfd\") pod \"console-6dc4c4ccd9-kv9l7\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") " pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.730124 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.730085    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8rlhj"
Aug 04 18:01:03.765440 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.765405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:03.792917 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.792887    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-fh84v" event={"ID":"f220e635-729a-4535-aa7a-27ffe2bc2fb1","Type":"ContainerStarted","Data":"f46c50d4c3fbf978c8cd0ec55ad03847bd7aa851ac0c4aae92a28ac66112d3e6"}
Aug 04 18:01:03.793159 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.793140    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-fh84v"
Aug 04 18:01:03.797948 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.797920    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-fh84v"
Aug 04 18:01:03.832749 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.832696    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-fh84v" podStartSLOduration=2.515496939 podStartE2EDuration="3.832678239s" totalImagesPullingTime="1.3171813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:03.812213802 +0000 UTC m=+63.788045498" watchObservedRunningTime="2026-08-04 18:01:03.832678239 +0000 UTC m=+63.808509930"
Aug 04 18:01:03.969289 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:03.969211    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.069370 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.069332    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.318483 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.318450    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dc4c4ccd9-kv9l7"]
Aug 04 18:01:04.321906 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:04.321877    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716abfd2_ee03_4ebc_bb89_d41d82d26090.slice/crio-d53ac1d0383c895572a9cdfb58ca82426328890cc130cba5bc18eb47615207d6 WatchSource:0}: Error finding container d53ac1d0383c895572a9cdfb58ca82426328890cc130cba5bc18eb47615207d6: Status 404 returned error can't find the container with id d53ac1d0383c895572a9cdfb58ca82426328890cc130cba5bc18eb47615207d6
Aug 04 18:01:04.565080 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.565042    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.600416 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.600387    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-frrgr"]
Aug 04 18:01:04.602685 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.602669    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:04.606340 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.606317    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Aug 04 18:01:04.606340 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.606328    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 04 18:01:04.606557 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.606383    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 04 18:01:04.606616 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.606559    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 04 18:01:04.606675 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.606664    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Aug 04 18:01:04.606890 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.606877    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rwlpb\""
Aug 04 18:01:04.613436 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.613416    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-frrgr"]
Aug 04 18:01:04.671972 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.671932    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.794763 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.794729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a21274d-e369-4eb0-bc41-11d724999623-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-frrgr\" (UID: \"1a21274d-e369-4eb0-bc41-11d724999623\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:04.794763 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.794762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a21274d-e369-4eb0-bc41-11d724999623-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-frrgr\" (UID: \"1a21274d-e369-4eb0-bc41-11d724999623\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:04.795008 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.794823    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a21274d-e369-4eb0-bc41-11d724999623-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-frrgr\" (UID: \"1a21274d-e369-4eb0-bc41-11d724999623\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:04.795008 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.794883    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4fw\" (UniqueName: \"kubernetes.io/projected/1a21274d-e369-4eb0-bc41-11d724999623-kube-api-access-bx4fw\") pod \"prometheus-operator-758d7cb874-frrgr\" (UID: \"1a21274d-e369-4eb0-bc41-11d724999623\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:04.798631 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.798599    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4m69" event={"ID":"25b94f13-bec2-41c1-a848-37a3d18170a9","Type":"ContainerStarted","Data":"f90c8122f1fac5e774d7b74351a1c53b9ccf494ce1cf029a6d4bfd9694491927"}
Aug 04 18:01:04.800087 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.800066    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-27kng" event={"ID":"3155211c-54d6-4956-9506-85213a63de71","Type":"ContainerStarted","Data":"78b21060986d426a5fcaada769191a0ce6b7269498b604b625b6bed65d5c6046"}
Aug 04 18:01:04.801238 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.801217    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc4c4ccd9-kv9l7" event={"ID":"716abfd2-ee03-4ebc-bb89-d41d82d26090","Type":"ContainerStarted","Data":"d53ac1d0383c895572a9cdfb58ca82426328890cc130cba5bc18eb47615207d6"}
Aug 04 18:01:04.819711 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.819626    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s4m69" podStartSLOduration=2.092938092 podStartE2EDuration="7.819612075s" totalImagesPullingTime="5.726673983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:04.818449122 +0000 UTC m=+64.794280822" watchObservedRunningTime="2026-08-04 18:01:04.819612075 +0000 UTC m=+64.795443771"
Aug 04 18:01:04.835697 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.835656    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-27kng" podStartSLOduration=2.054481778 podStartE2EDuration="7.835642772s" totalImagesPullingTime="5.781160994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:04.83486251 +0000 UTC m=+64.810694207" watchObservedRunningTime="2026-08-04 18:01:04.835642772 +0000 UTC m=+64.811474469"
Aug 04 18:01:04.895463 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.895422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a21274d-e369-4eb0-bc41-11d724999623-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-frrgr\" (UID: \"1a21274d-e369-4eb0-bc41-11d724999623\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:04.895648 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.895480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4fw\" (UniqueName: \"kubernetes.io/projected/1a21274d-e369-4eb0-bc41-11d724999623-kube-api-access-bx4fw\") pod \"prometheus-operator-758d7cb874-frrgr\" (UID: \"1a21274d-e369-4eb0-bc41-11d724999623\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:04.895648 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.895545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a21274d-e369-4eb0-bc41-11d724999623-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-frrgr\" (UID: \"1a21274d-e369-4eb0-bc41-11d724999623\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:04.895648 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.895569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a21274d-e369-4eb0-bc41-11d724999623-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-frrgr\" (UID: \"1a21274d-e369-4eb0-bc41-11d724999623\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:04.895818 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:01:04.895681    2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Aug 04 18:01:04.895818 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:01:04.895745    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a21274d-e369-4eb0-bc41-11d724999623-prometheus-operator-tls podName:1a21274d-e369-4eb0-bc41-11d724999623 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:05.395730657 +0000 UTC m=+65.371562337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/1a21274d-e369-4eb0-bc41-11d724999623-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-frrgr" (UID: "1a21274d-e369-4eb0-bc41-11d724999623") : secret "prometheus-operator-tls" not found
Aug 04 18:01:04.896296 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.896272    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a21274d-e369-4eb0-bc41-11d724999623-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-frrgr\" (UID: \"1a21274d-e369-4eb0-bc41-11d724999623\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:04.897728 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.897707    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a21274d-e369-4eb0-bc41-11d724999623-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-frrgr\" (UID: \"1a21274d-e369-4eb0-bc41-11d724999623\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:04.904461 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.904445    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4fw\" (UniqueName: \"kubernetes.io/projected/1a21274d-e369-4eb0-bc41-11d724999623-kube-api-access-bx4fw\") pod \"prometheus-operator-758d7cb874-frrgr\" (UID: \"1a21274d-e369-4eb0-bc41-11d724999623\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:04.958040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:04.958003    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:05.398986 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:05.398948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a21274d-e369-4eb0-bc41-11d724999623-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-frrgr\" (UID: \"1a21274d-e369-4eb0-bc41-11d724999623\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:05.401276 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:05.401246    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a21274d-e369-4eb0-bc41-11d724999623-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-frrgr\" (UID: \"1a21274d-e369-4eb0-bc41-11d724999623\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:05.516307 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:05.516271    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr"
Aug 04 18:01:05.639463 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:05.639426    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-frrgr"]
Aug 04 18:01:05.643538 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:05.643512    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a21274d_e369_4eb0_bc41_11d724999623.slice/crio-35e29ce08bae82554607d70d8866a2d5926b373b7a76bcf280f8d75bf8b7ab27 WatchSource:0}: Error finding container 35e29ce08bae82554607d70d8866a2d5926b373b7a76bcf280f8d75bf8b7ab27: Status 404 returned error can't find the container with id 35e29ce08bae82554607d70d8866a2d5926b373b7a76bcf280f8d75bf8b7ab27
Aug 04 18:01:05.805033 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:05.804996    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr" event={"ID":"1a21274d-e369-4eb0-bc41-11d724999623","Type":"ContainerStarted","Data":"35e29ce08bae82554607d70d8866a2d5926b373b7a76bcf280f8d75bf8b7ab27"}
Aug 04 18:01:06.114252 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.114176    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:06.180958 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.180923    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:06.205312 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.205285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:01:06.208491 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.208475    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:01:06.218065 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.218041    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9274e13-87e6-4bb8-bb29-3ed6fcde80e0-original-pull-secret\") pod \"global-pull-secret-syncer-6kdvz\" (UID: \"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0\") " pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:01:06.306279 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.306245    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs\") pod \"network-metrics-daemon-7ggnb\" (UID: \"62165010-2435-476c-bf99-75bf927b2b45\") " pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:01:06.309496 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.309477    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:01:06.318860 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.318840    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62165010-2435-476c-bf99-75bf927b2b45-metrics-certs\") pod \"network-metrics-daemon-7ggnb\" (UID: \"62165010-2435-476c-bf99-75bf927b2b45\") " pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:01:06.407325 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.407241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsxm\" (UniqueName: \"kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm\") pod \"network-check-target-q5kpw\" (UID: \"110d78d6-4775-4717-84e3-030a8c109a9b\") " pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:01:06.410421 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.410404    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 18:01:06.420392 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.420370    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 18:01:06.430886 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.430866    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmsxm\" (UniqueName: \"kubernetes.io/projected/110d78d6-4775-4717-84e3-030a8c109a9b-kube-api-access-zmsxm\") pod \"network-check-target-q5kpw\" (UID: \"110d78d6-4775-4717-84e3-030a8c109a9b\") " pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:01:06.490223 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.490200    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qxtnm\""
Aug 04 18:01:06.494265 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.494251    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6kdvz"
Aug 04 18:01:06.498827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.498808    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:01:06.502325 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.502309    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-88fc8\""
Aug 04 18:01:06.510537 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.510512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7ggnb"
Aug 04 18:01:06.630509 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.630480    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6kdvz"]
Aug 04 18:01:06.633646 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:06.633610    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9274e13_87e6_4bb8_bb29_3ed6fcde80e0.slice/crio-83635d531cae2e5229d1f7d71c1cd0b2a888031beb8808cdf2bfe3985b63a497 WatchSource:0}: Error finding container 83635d531cae2e5229d1f7d71c1cd0b2a888031beb8808cdf2bfe3985b63a497: Status 404 returned error can't find the container with id 83635d531cae2e5229d1f7d71c1cd0b2a888031beb8808cdf2bfe3985b63a497
Aug 04 18:01:06.651315 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.651288    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q5kpw"]
Aug 04 18:01:06.655270 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:06.655247    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110d78d6_4775_4717_84e3_030a8c109a9b.slice/crio-81e125a2837e6efbba6be1eb7a3c73598261d449016619cf14e2148783cc8f62 WatchSource:0}: Error finding container 81e125a2837e6efbba6be1eb7a3c73598261d449016619cf14e2148783cc8f62: Status 404 returned error can't find the container with id 81e125a2837e6efbba6be1eb7a3c73598261d449016619cf14e2148783cc8f62
Aug 04 18:01:06.669259 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.669234    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7ggnb"]
Aug 04 18:01:06.672599 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:06.672575    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62165010_2435_476c_bf99_75bf927b2b45.slice/crio-8faf766ba3c807550693275ea59e43c72d4ee3580b5bec776df188d984be9382 WatchSource:0}: Error finding container 8faf766ba3c807550693275ea59e43c72d4ee3580b5bec776df188d984be9382: Status 404 returned error can't find the container with id 8faf766ba3c807550693275ea59e43c72d4ee3580b5bec776df188d984be9382
Aug 04 18:01:06.808626 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.808588    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7ggnb" event={"ID":"62165010-2435-476c-bf99-75bf927b2b45","Type":"ContainerStarted","Data":"8faf766ba3c807550693275ea59e43c72d4ee3580b5bec776df188d984be9382"}
Aug 04 18:01:06.809575 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.809546    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q5kpw" event={"ID":"110d78d6-4775-4717-84e3-030a8c109a9b","Type":"ContainerStarted","Data":"81e125a2837e6efbba6be1eb7a3c73598261d449016619cf14e2148783cc8f62"}
Aug 04 18:01:06.810672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:06.810651    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6kdvz" event={"ID":"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0","Type":"ContainerStarted","Data":"83635d531cae2e5229d1f7d71c1cd0b2a888031beb8808cdf2bfe3985b63a497"}
Aug 04 18:01:07.309782 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:07.309693    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:07.408965 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:07.408934    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:08.224712 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:08.224560    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:08.383028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:08.382963    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:08.824266 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:08.824226    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7ggnb" event={"ID":"62165010-2435-476c-bf99-75bf927b2b45","Type":"ContainerStarted","Data":"d056fe046d949eb980c70611f8fa1902d677709d8f5901a8dd6a91217d94882d"}
Aug 04 18:01:08.824718 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:08.824299    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7ggnb" event={"ID":"62165010-2435-476c-bf99-75bf927b2b45","Type":"ContainerStarted","Data":"67bdafe72232b46c0c852472a50322759d15d317e9147337afa575e6bacafbe0"}
Aug 04 18:01:08.846943 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:08.845964    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7ggnb" podStartSLOduration=66.562489833 podStartE2EDuration="1m7.845943854s" totalImagesPullingTime="1.283454021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:08.844226919 +0000 UTC m=+68.820058617" watchObservedRunningTime="2026-08-04 18:01:08.845943854 +0000 UTC m=+68.821775556"
Aug 04 18:01:08.958417 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:08.958316    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:09.054500 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:09.054460    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.780779 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:10.780745    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.788730 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:10.788709    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w7w5n"
Aug 04 18:01:10.884020 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:10.883970    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.984620 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:10.984574    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:11.711734 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:11.711676    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:11.780932 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:11.780897    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:12.504765 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.504729    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d55549997-cj5gb"]
Aug 04 18:01:12.508939 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.508915    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.517351 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.517326    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Aug 04 18:01:12.517770 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.517676    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:12.517895 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.517861    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d55549997-cj5gb"]
Aug 04 18:01:12.554091 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.554056    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-service-ca\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.554091 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.554099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-serving-cert\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.554322 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.554140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhm4v\" (UniqueName: \"kubernetes.io/projected/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-kube-api-access-hhm4v\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.554322 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.554173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-oauth-config\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.554322 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.554199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-config\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.554322 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.554310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-oauth-serving-cert\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.554459 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.554357    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-trusted-ca-bundle\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.582058 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.582031    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:12.655348 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.655315    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-oauth-serving-cert\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.655540 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.655373    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-trusted-ca-bundle\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.655540 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.655406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-service-ca\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.655540 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.655434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-serving-cert\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.655540 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.655457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhm4v\" (UniqueName: \"kubernetes.io/projected/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-kube-api-access-hhm4v\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.655540 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.655487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-oauth-config\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.655540 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.655522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-config\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.656231 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.656178    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-oauth-serving-cert\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.656231 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.656195    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-config\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.656455 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.656397    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-trusted-ca-bundle\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.656619 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.656594    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-service-ca\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.658288 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.658261    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-oauth-config\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.658448 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.658428    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-serving-cert\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.666613 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.666585    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhm4v\" (UniqueName: \"kubernetes.io/projected/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-kube-api-access-hhm4v\") pod \"console-d55549997-cj5gb\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") " pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.819842 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.819815    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:12.982494 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:12.982444    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d55549997-cj5gb"]
Aug 04 18:01:12.985421 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:12.985390    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4e3b5a_c43a_4b19_b7fe_3478a6b9b623.slice/crio-8bd62201952cf32eda23eacccb22a3fceface268b95aa4b301a35a5a901a68c1 WatchSource:0}: Error finding container 8bd62201952cf32eda23eacccb22a3fceface268b95aa4b301a35a5a901a68c1: Status 404 returned error can't find the container with id 8bd62201952cf32eda23eacccb22a3fceface268b95aa4b301a35a5a901a68c1
Aug 04 18:01:13.108911 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:13.108875    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:13.848951 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:13.848914    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d55549997-cj5gb" event={"ID":"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623","Type":"ContainerStarted","Data":"3d22bf370ea8a798beb5673921a6cb01039feb5723d0f2e28e624306811abd0f"}
Aug 04 18:01:13.849439 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:13.848971    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d55549997-cj5gb" event={"ID":"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623","Type":"ContainerStarted","Data":"8bd62201952cf32eda23eacccb22a3fceface268b95aa4b301a35a5a901a68c1"}
Aug 04 18:01:13.850674 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:13.850651    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr" event={"ID":"1a21274d-e369-4eb0-bc41-11d724999623","Type":"ContainerStarted","Data":"a55a7d8db7715c910786ad37ca5122a2abf319b32ebdc6a9b3089fa3ef096250"}
Aug 04 18:01:13.850798 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:13.850704    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr" event={"ID":"1a21274d-e369-4eb0-bc41-11d724999623","Type":"ContainerStarted","Data":"6c488a6e57b505dbeee9485caffc680647d9524af4199fcfd57cb6c3efaa5b96"}
Aug 04 18:01:13.851994 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:13.851972    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q5kpw" event={"ID":"110d78d6-4775-4717-84e3-030a8c109a9b","Type":"ContainerStarted","Data":"54c14ec2f478c22434a544c8a38a5c54bb05566e4a062522bd44e15a41bf05f3"}
Aug 04 18:01:13.852071 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:13.852030    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:01:13.853463 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:13.853443    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6kdvz" event={"ID":"e9274e13-87e6-4bb8-bb29-3ed6fcde80e0","Type":"ContainerStarted","Data":"f0623b1f2c823d679d906f86e8d85165025d7e4890006b776f8da3cdda751f4a"}
Aug 04 18:01:13.854861 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:13.854842    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc4c4ccd9-kv9l7" event={"ID":"716abfd2-ee03-4ebc-bb89-d41d82d26090","Type":"ContainerStarted","Data":"b28221a866ff55b85444968db2483f55c114b83771f5239b586d9249ade1f368"}
Aug 04 18:01:13.867289 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:13.867249    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d55549997-cj5gb" podStartSLOduration=1.867236017 podStartE2EDuration="1.867236017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:13.866810919 +0000 UTC m=+73.842642609" watchObservedRunningTime="2026-08-04 18:01:13.867236017 +0000 UTC m=+73.843067692"
Aug 04 18:01:13.881390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:13.881336    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6kdvz" podStartSLOduration=66.734295309 podStartE2EDuration="1m12.881318356s" totalImagesPullingTime="6.147023047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:13.881300454 +0000 UTC m=+73.857132151" watchObservedRunningTime="2026-08-04 18:01:13.881318356 +0000 UTC m=+73.857150054"
Aug 04 18:01:13.901841 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:13.901788    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q5kpw" podStartSLOduration=66.776235085 podStartE2EDuration="1m12.9017718s" totalImagesPullingTime="6.125536715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:13.901472321 +0000 UTC m=+73.877304019" watchObservedRunningTime="2026-08-04 18:01:13.9017718 +0000 UTC m=+73.877603497"
Aug 04 18:01:13.920367 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:13.920311    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dc4c4ccd9-kv9l7" podStartSLOduration=2.475551104 podStartE2EDuration="10.92029544s" totalImagesPullingTime="8.444744336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:13.919680544 +0000 UTC m=+73.895512242" watchObservedRunningTime="2026-08-04 18:01:13.92029544 +0000 UTC m=+73.896127205"
Aug 04 18:01:13.937490 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:13.937444    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-frrgr" podStartSLOduration=2.814322454 podStartE2EDuration="9.937430941s" totalImagesPullingTime="7.123108487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:13.936578786 +0000 UTC m=+73.912410503" watchObservedRunningTime="2026-08-04 18:01:13.937430941 +0000 UTC m=+73.913262638"
Aug 04 18:01:15.721741 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:15.721706    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:15.829280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:15.829243    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:15.987263 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:15.987166    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:16.022394 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.022365    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"]
Aug 04 18:01:16.024436 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.024418    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.027305 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.027284    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Aug 04 18:01:16.027851 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.027826    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Aug 04 18:01:16.028241 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.028228    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-pv68z\""
Aug 04 18:01:16.032240 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.032219    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"]
Aug 04 18:01:16.034323 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.034305    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.036754 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.036730    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Aug 04 18:01:16.036853 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.036791    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Aug 04 18:01:16.036853 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.036798    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Aug 04 18:01:16.036853 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.036793    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-b4lzt\""
Aug 04 18:01:16.041807 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.041785    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"]
Aug 04 18:01:16.048491 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.048468    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"]
Aug 04 18:01:16.050991 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.050971    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9rrr8"]
Aug 04 18:01:16.053231 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.053211    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.055496 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.055472    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 04 18:01:16.055607 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.055595    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 04 18:01:16.055728 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.055670    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 04 18:01:16.055728 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.055698    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2sv5j\""
Aug 04 18:01:16.083373 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.083373 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083371    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-accelerators-collector-config\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.083589 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083399    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-textfile\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.083589 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083445    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5bfce57-8a11-487c-8840-53bdd34ec60a-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-mw69r\" (UID: \"c5bfce57-8a11-487c-8840-53bdd34ec60a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.083589 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.083589 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a0ec7b7e-324b-4450-b564-78772ca59e95-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.083589 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7pr4\" (UniqueName: \"kubernetes.io/projected/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-api-access-p7pr4\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.083589 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083525    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7c755fd3-d8ce-4de8-9348-251d30952904-root\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.083589 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083548    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c755fd3-d8ce-4de8-9348-251d30952904-sys\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.083589 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5bfce57-8a11-487c-8840-53bdd34ec60a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-mw69r\" (UID: \"c5bfce57-8a11-487c-8840-53bdd34ec60a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.083589 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-wtmp\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.083887 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqzdl\" (UniqueName: \"kubernetes.io/projected/7c755fd3-d8ce-4de8-9348-251d30952904-kube-api-access-wqzdl\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.083887 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083674    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs2sb\" (UniqueName: \"kubernetes.io/projected/c5bfce57-8a11-487c-8840-53bdd34ec60a-kube-api-access-cs2sb\") pod \"openshift-state-metrics-6bb4d7b4dd-mw69r\" (UID: \"c5bfce57-8a11-487c-8840-53bdd34ec60a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.083887 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5bfce57-8a11-487c-8840-53bdd34ec60a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-mw69r\" (UID: \"c5bfce57-8a11-487c-8840-53bdd34ec60a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.083887 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083728    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0ec7b7e-324b-4450-b564-78772ca59e95-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.083887 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083777    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c755fd3-d8ce-4de8-9348-251d30952904-metrics-client-ca\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.083887 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083801    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-tls\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.083887 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083820    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.083887 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.083844    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.090468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.090442    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:16.181265 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.181219    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:16.184556 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.184530    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.184702 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.184566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.184702 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.184609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.184702 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.184632    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-accelerators-collector-config\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.184702 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.184658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-textfile\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.184916 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.184811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5bfce57-8a11-487c-8840-53bdd34ec60a-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-mw69r\" (UID: \"c5bfce57-8a11-487c-8840-53bdd34ec60a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.184916 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.184857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.184916 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.184889    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a0ec7b7e-324b-4450-b564-78772ca59e95-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.185067 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.184918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7pr4\" (UniqueName: \"kubernetes.io/projected/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-api-access-p7pr4\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.185067 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.184941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7c755fd3-d8ce-4de8-9348-251d30952904-root\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.185067 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.184980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c755fd3-d8ce-4de8-9348-251d30952904-sys\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.185067 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185011    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5bfce57-8a11-487c-8840-53bdd34ec60a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-mw69r\" (UID: \"c5bfce57-8a11-487c-8840-53bdd34ec60a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.185067 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-wtmp\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.185067 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185062    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-textfile\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.185390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185077    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqzdl\" (UniqueName: \"kubernetes.io/projected/7c755fd3-d8ce-4de8-9348-251d30952904-kube-api-access-wqzdl\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.185390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs2sb\" (UniqueName: \"kubernetes.io/projected/c5bfce57-8a11-487c-8840-53bdd34ec60a-kube-api-access-cs2sb\") pod \"openshift-state-metrics-6bb4d7b4dd-mw69r\" (UID: \"c5bfce57-8a11-487c-8840-53bdd34ec60a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.185390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185161    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7c755fd3-d8ce-4de8-9348-251d30952904-root\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.185390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5bfce57-8a11-487c-8840-53bdd34ec60a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-mw69r\" (UID: \"c5bfce57-8a11-487c-8840-53bdd34ec60a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.185390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185228    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0ec7b7e-324b-4450-b564-78772ca59e95-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.185390 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:01:16.185242    2576 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Aug 04 18:01:16.185390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c755fd3-d8ce-4de8-9348-251d30952904-metrics-client-ca\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.185390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-tls\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.185390 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:01:16.185313    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5bfce57-8a11-487c-8840-53bdd34ec60a-openshift-state-metrics-tls podName:c5bfce57-8a11-487c-8840-53bdd34ec60a nodeName:}" failed. No retries permitted until 2026-08-04 18:01:16.685293656 +0000 UTC m=+76.661125333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c5bfce57-8a11-487c-8840-53bdd34ec60a-openshift-state-metrics-tls") pod "openshift-state-metrics-6bb4d7b4dd-mw69r" (UID: "c5bfce57-8a11-487c-8840-53bdd34ec60a") : secret "openshift-state-metrics-tls" not found
Aug 04 18:01:16.185390 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:01:16.185368    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Aug 04 18:01:16.185885 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185405    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c755fd3-d8ce-4de8-9348-251d30952904-sys\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.185885 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:01:16.185412    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-tls podName:7c755fd3-d8ce-4de8-9348-251d30952904 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:16.685397017 +0000 UTC m=+76.661228693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-tls") pod "node-exporter-9rrr8" (UID: "7c755fd3-d8ce-4de8-9348-251d30952904") : secret "node-exporter-tls" not found
Aug 04 18:01:16.185885 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:01:16.185249    2576 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Aug 04 18:01:16.185885 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:01:16.185467    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-state-metrics-tls podName:a0ec7b7e-324b-4450-b564-78772ca59e95 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:16.685455336 +0000 UTC m=+76.661287013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-state-metrics-tls") pod "kube-state-metrics-6bd9664c4-bsx9g" (UID: "a0ec7b7e-324b-4450-b564-78772ca59e95") : secret "kube-state-metrics-tls" not found
Aug 04 18:01:16.185885 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185476    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.185885 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185525    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5bfce57-8a11-487c-8840-53bdd34ec60a-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-mw69r\" (UID: \"c5bfce57-8a11-487c-8840-53bdd34ec60a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.185885 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185568    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a0ec7b7e-324b-4450-b564-78772ca59e95-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.185885 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.185635    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-wtmp\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.186293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.186087    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-accelerators-collector-config\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.186293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.186130    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c755fd3-d8ce-4de8-9348-251d30952904-metrics-client-ca\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.186293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.186191    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0ec7b7e-324b-4450-b564-78772ca59e95-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.187505 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.187482    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.187733 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.187717    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5bfce57-8a11-487c-8840-53bdd34ec60a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-mw69r\" (UID: \"c5bfce57-8a11-487c-8840-53bdd34ec60a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.187816 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.187799    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.194332 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.194310    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqzdl\" (UniqueName: \"kubernetes.io/projected/7c755fd3-d8ce-4de8-9348-251d30952904-kube-api-access-wqzdl\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.194433 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.194311    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs2sb\" (UniqueName: \"kubernetes.io/projected/c5bfce57-8a11-487c-8840-53bdd34ec60a-kube-api-access-cs2sb\") pod \"openshift-state-metrics-6bb4d7b4dd-mw69r\" (UID: \"c5bfce57-8a11-487c-8840-53bdd34ec60a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.194433 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.194355    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7pr4\" (UniqueName: \"kubernetes.io/projected/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-api-access-p7pr4\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.302851 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.302818    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:16.402456 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.402413    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:16.688921 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.688834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.688921 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.688881    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5bfce57-8a11-487c-8840-53bdd34ec60a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-mw69r\" (UID: \"c5bfce57-8a11-487c-8840-53bdd34ec60a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.688921 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.688919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-tls\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.691221 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.691196    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7c755fd3-d8ce-4de8-9348-251d30952904-node-exporter-tls\") pod \"node-exporter-9rrr8\" (UID: \"7c755fd3-d8ce-4de8-9348-251d30952904\") " pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.691346 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.691244    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5bfce57-8a11-487c-8840-53bdd34ec60a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-mw69r\" (UID: \"c5bfce57-8a11-487c-8840-53bdd34ec60a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.691346 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.691243    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0ec7b7e-324b-4450-b564-78772ca59e95-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-bsx9g\" (UID: \"a0ec7b7e-324b-4450-b564-78772ca59e95\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.934156 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.934094    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"
Aug 04 18:01:16.942910 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.942851    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"
Aug 04 18:01:16.964212 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:16.964186    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9rrr8"
Aug 04 18:01:16.976483 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:16.976448    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c755fd3_d8ce_4de8_9348_251d30952904.slice/crio-6143a364a13d5e4e7d8ba42964897c8051b6584f357d2564772c7371c09c186c WatchSource:0}: Error finding container 6143a364a13d5e4e7d8ba42964897c8051b6584f357d2564772c7371c09c186c: Status 404 returned error can't find the container with id 6143a364a13d5e4e7d8ba42964897c8051b6584f357d2564772c7371c09c186c
Aug 04 18:01:17.088885 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.088853    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r"]
Aug 04 18:01:17.092357 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:17.092322    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5bfce57_8a11_487c_8840_53bdd34ec60a.slice/crio-37a6e1c9e4383c4c02fdb9df01692608c3012e48240bdbce66281657742f7aa5 WatchSource:0}: Error finding container 37a6e1c9e4383c4c02fdb9df01692608c3012e48240bdbce66281657742f7aa5: Status 404 returned error can't find the container with id 37a6e1c9e4383c4c02fdb9df01692608c3012e48240bdbce66281657742f7aa5
Aug 04 18:01:17.107912 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.107891    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g"]
Aug 04 18:01:17.112230 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:17.112207    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ec7b7e_324b_4450_b564_78772ca59e95.slice/crio-a17f303f3784950107c0b21c09792a69ff895c41b3f9c836f1f581e04c266f76 WatchSource:0}: Error finding container a17f303f3784950107c0b21c09792a69ff895c41b3f9c836f1f581e04c266f76: Status 404 returned error can't find the container with id a17f303f3784950107c0b21c09792a69ff895c41b3f9c836f1f581e04c266f76
Aug 04 18:01:17.162695 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.162669    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:01:17.165289 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.165268    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.169159 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.169101    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Aug 04 18:01:17.169287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.169264    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Aug 04 18:01:17.169358 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.169300    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Aug 04 18:01:17.169672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.169651    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Aug 04 18:01:17.174329 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.170362    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Aug 04 18:01:17.174329 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.170421    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Aug 04 18:01:17.174329 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.170550    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Aug 04 18:01:17.174329 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.170564    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Aug 04 18:01:17.174329 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.171072    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Aug 04 18:01:17.174329 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.171464    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d865b\""
Aug 04 18:01:17.188351 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.188278    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:01:17.192988 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.192933    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa34905-832c-4e14-a6b0-77304fc88782-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.192988 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.192983    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s8z5\" (UniqueName: \"kubernetes.io/projected/9aa34905-832c-4e14-a6b0-77304fc88782-kube-api-access-5s8z5\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.193171 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.193043    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa34905-832c-4e14-a6b0-77304fc88782-config-out\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.193171 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.193147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.193283 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.193187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.193337 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.193289    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa34905-832c-4e14-a6b0-77304fc88782-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.193399 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.193357    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-config-volume\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.193452 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.193420    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.193500 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.193453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.193500 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.193478    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-web-config\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.193576 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.193518    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aa34905-832c-4e14-a6b0-77304fc88782-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.193619 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.193576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.193619 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.193603    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa34905-832c-4e14-a6b0-77304fc88782-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.278407 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.278375    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:17.294130 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.294076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa34905-832c-4e14-a6b0-77304fc88782-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.294298 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.294136    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s8z5\" (UniqueName: \"kubernetes.io/projected/9aa34905-832c-4e14-a6b0-77304fc88782-kube-api-access-5s8z5\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.294298 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.294163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa34905-832c-4e14-a6b0-77304fc88782-config-out\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.294298 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.294185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.294298 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.294208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.294298 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.294276    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa34905-832c-4e14-a6b0-77304fc88782-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.294587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.294313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-config-volume\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.294587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.294352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.294587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.294382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.294587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.294406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-web-config\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.294587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.294432    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aa34905-832c-4e14-a6b0-77304fc88782-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.294587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.294474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.294587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.294476    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa34905-832c-4e14-a6b0-77304fc88782-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.294587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.294500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa34905-832c-4e14-a6b0-77304fc88782-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.295063 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.295036    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa34905-832c-4e14-a6b0-77304fc88782-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.296309 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.296262    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aa34905-832c-4e14-a6b0-77304fc88782-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.297385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.297357    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa34905-832c-4e14-a6b0-77304fc88782-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.297461 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.297445    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-config-volume\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.297595 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.297563    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-web-config\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.297738 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.297717    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.297822 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.297792    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.298316 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.298296    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.298578 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.298558    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa34905-832c-4e14-a6b0-77304fc88782-config-out\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.298683 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.298668    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.298946 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.298925    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.302105 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.302085    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s8z5\" (UniqueName: \"kubernetes.io/projected/9aa34905-832c-4e14-a6b0-77304fc88782-kube-api-access-5s8z5\") pod \"alertmanager-main-0\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.376581 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.376535    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:17.503205 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.503104    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:17.676068 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.676041    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:01:17.872740 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.872613    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rrr8" event={"ID":"7c755fd3-d8ce-4de8-9348-251d30952904","Type":"ContainerStarted","Data":"6143a364a13d5e4e7d8ba42964897c8051b6584f357d2564772c7371c09c186c"}
Aug 04 18:01:17.874305 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.874273    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g" event={"ID":"a0ec7b7e-324b-4450-b564-78772ca59e95","Type":"ContainerStarted","Data":"a17f303f3784950107c0b21c09792a69ff895c41b3f9c836f1f581e04c266f76"}
Aug 04 18:01:17.877271 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.877240    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r" event={"ID":"c5bfce57-8a11-487c-8840-53bdd34ec60a","Type":"ContainerStarted","Data":"7092acd6737432b904d1d4404f90c2f9d73fdc12be964e8a6874bd38b884400e"}
Aug 04 18:01:17.877383 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.877298    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r" event={"ID":"c5bfce57-8a11-487c-8840-53bdd34ec60a","Type":"ContainerStarted","Data":"b73fd08f6b126cc7416a69211d00f580728ace1879161e30a7b1cd69ff62e10d"}
Aug 04 18:01:17.877383 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.877326    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r" event={"ID":"c5bfce57-8a11-487c-8840-53bdd34ec60a","Type":"ContainerStarted","Data":"37a6e1c9e4383c4c02fdb9df01692608c3012e48240bdbce66281657742f7aa5"}
Aug 04 18:01:17.900798 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:17.900764    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa34905_832c_4e14_a6b0_77304fc88782.slice/crio-248c563f97077ae6bbdd7724e3b3f5f91b2823c469ff302e4b2345eaf27c50d1 WatchSource:0}: Error finding container 248c563f97077ae6bbdd7724e3b3f5f91b2823c469ff302e4b2345eaf27c50d1: Status 404 returned error can't find the container with id 248c563f97077ae6bbdd7724e3b3f5f91b2823c469ff302e4b2345eaf27c50d1
Aug 04 18:01:17.919638 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:17.919480    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:18.019346 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:18.019305    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:18.884373 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:18.884210    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g" event={"ID":"a0ec7b7e-324b-4450-b564-78772ca59e95","Type":"ContainerStarted","Data":"34d685b45749f47f9c6a8564b37259d7fe6eaf3c974f9b3fae34d7c158df95b1"}
Aug 04 18:01:18.884373 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:18.884278    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g" event={"ID":"a0ec7b7e-324b-4450-b564-78772ca59e95","Type":"ContainerStarted","Data":"c92b351becc9a36a961b07c42c72ef575e68838c1e3697b1583ad9d264d8da98"}
Aug 04 18:01:18.884373 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:18.884300    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g" event={"ID":"a0ec7b7e-324b-4450-b564-78772ca59e95","Type":"ContainerStarted","Data":"01cb5f155b1fe0922a8828eb7172b9e49e7833bd1e78977c89ff3603fe56e5ce"}
Aug 04 18:01:18.886782 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:18.886757    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r" event={"ID":"c5bfce57-8a11-487c-8840-53bdd34ec60a","Type":"ContainerStarted","Data":"367020c97423de5f83b2c18299c3d3194fe4c7d935ceb150c972a9e1f064db93"}
Aug 04 18:01:18.888223 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:18.888196    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerStarted","Data":"248c563f97077ae6bbdd7724e3b3f5f91b2823c469ff302e4b2345eaf27c50d1"}
Aug 04 18:01:18.889954 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:18.889924    2576 generic.go:356] "Generic (PLEG): container finished" podID="7c755fd3-d8ce-4de8-9348-251d30952904" containerID="668bc175cc5f99d0ce0ba0a02d0613399d66293f5e278b43c0ce39ef28938d9f" exitCode=0
Aug 04 18:01:18.890061 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:18.889978    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rrr8" event={"ID":"7c755fd3-d8ce-4de8-9348-251d30952904","Type":"ContainerDied","Data":"668bc175cc5f99d0ce0ba0a02d0613399d66293f5e278b43c0ce39ef28938d9f"}
Aug 04 18:01:18.911572 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:18.911503    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-bsx9g" podStartSLOduration=1.5747323770000001 podStartE2EDuration="2.911482671s" totalImagesPullingTime="1.336750294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:18.911018289 +0000 UTC m=+78.886849992" watchObservedRunningTime="2026-08-04 18:01:18.911482671 +0000 UTC m=+78.887314369"
Aug 04 18:01:18.960802 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:18.960614    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-mw69r" podStartSLOduration=1.744790088 podStartE2EDuration="2.960593602s" totalImagesPullingTime="1.215803514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:18.958961694 +0000 UTC m=+78.934793392" watchObservedRunningTime="2026-08-04 18:01:18.960593602 +0000 UTC m=+78.936425300"
Aug 04 18:01:18.987506 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:18.987412    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:19.069947 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.069905    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c679f8c78-2rshq"]
Aug 04 18:01:19.073548 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.073526    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.076274 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.076249    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2lbbj\""
Aug 04 18:01:19.076385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.076249    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7o4r8abipeolh\""
Aug 04 18:01:19.076385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.076303    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Aug 04 18:01:19.076734 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.076693    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Aug 04 18:01:19.076848 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.076748    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Aug 04 18:01:19.076848 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.076813    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Aug 04 18:01:19.076959 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.076882    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Aug 04 18:01:19.104791 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.104765    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c679f8c78-2rshq"]
Aug 04 18:01:19.108787 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.108762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch9v7\" (UniqueName: \"kubernetes.io/projected/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-kube-api-access-ch9v7\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.108908 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.108801    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.108956 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.108911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-metrics-client-ca\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.108956 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.108948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-grpc-tls\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.109028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.109014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.109070 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.109039    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.109128 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.109064    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-tls\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.109128 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.109087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.128035 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.127989    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:19.210170 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.210139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-metrics-client-ca\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.210170 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.210173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-grpc-tls\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.210363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.210199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.210363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.210224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.210363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.210253    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-tls\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.210363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.210279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.210363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.210350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch9v7\" (UniqueName: \"kubernetes.io/projected/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-kube-api-access-ch9v7\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.210599 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.210379    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.210953 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.210900    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-metrics-client-ca\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.212780 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.212751    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.212994 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.212971    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.213440 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.213406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.213440 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.213419    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.213440 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.213433    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-grpc-tls\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.213919 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.213896    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-secret-thanos-querier-tls\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.219657 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.219637    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch9v7\" (UniqueName: \"kubernetes.io/projected/8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4-kube-api-access-ch9v7\") pod \"thanos-querier-7c679f8c78-2rshq\" (UID: \"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4\") " pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.268734 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.268702    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:19.385169 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.385106    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:19.420051 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.420010    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:19.541918 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.541874    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c679f8c78-2rshq"]
Aug 04 18:01:19.544274 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:19.544248    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8222a7d2_3a06_4ae1_9f5a_9ccfc9da28d4.slice/crio-adbc3c659aca2ead9cf46c10b58ea4604d893487acdd0b6850423dd7ecc6ab9d WatchSource:0}: Error finding container adbc3c659aca2ead9cf46c10b58ea4604d893487acdd0b6850423dd7ecc6ab9d: Status 404 returned error can't find the container with id adbc3c659aca2ead9cf46c10b58ea4604d893487acdd0b6850423dd7ecc6ab9d
Aug 04 18:01:19.606906 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.606873    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:19.694497 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.694458    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:19.759026 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.758942    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:19.896175 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.896137    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rrr8" event={"ID":"7c755fd3-d8ce-4de8-9348-251d30952904","Type":"ContainerStarted","Data":"f5e90f7f97ac3ad369ebe27b966be096c9e925aac3c4870b447c2c0919339345"}
Aug 04 18:01:19.896346 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.896199    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rrr8" event={"ID":"7c755fd3-d8ce-4de8-9348-251d30952904","Type":"ContainerStarted","Data":"77bbbc11ac3633128eed8225dab10c247c80bf3443ec8462d3a17a2f84fea413"}
Aug 04 18:01:19.897391 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.897364    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq" event={"ID":"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4","Type":"ContainerStarted","Data":"adbc3c659aca2ead9cf46c10b58ea4604d893487acdd0b6850423dd7ecc6ab9d"}
Aug 04 18:01:19.898732 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.898709    2576 generic.go:356] "Generic (PLEG): container finished" podID="9aa34905-832c-4e14-a6b0-77304fc88782" containerID="504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e" exitCode=0
Aug 04 18:01:19.898823 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.898788    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerDied","Data":"504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e"}
Aug 04 18:01:19.916985 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:19.916935    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9rrr8" podStartSLOduration=2.348271777 podStartE2EDuration="3.916919924s" totalImagesPullingTime="1.468981043s" totalInitContainerRuntime="99.667104ms" isStatefulPod=true podCreationTimestamp="2026-08-04 18:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:19.915328517 +0000 UTC m=+79.891160216" watchObservedRunningTime="2026-08-04 18:01:19.916919924 +0000 UTC m=+79.892751621"
Aug 04 18:01:20.343811 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.343776    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-678f56c7df-qv25h"]
Aug 04 18:01:20.345989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.345974    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.349376 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.349260    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Aug 04 18:01:20.349376 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.349306    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Aug 04 18:01:20.349555 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.349385    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Aug 04 18:01:20.349555 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.349416    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Aug 04 18:01:20.350463 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.350447    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3bs8f3laojm5j\""
Aug 04 18:01:20.350539 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.350504    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rjdsb\""
Aug 04 18:01:20.357674 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.357654    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-678f56c7df-qv25h"]
Aug 04 18:01:20.419960 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.419927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e7784b-6a83-4397-ad9e-aae932de61b9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.419960 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.419963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/79e7784b-6a83-4397-ad9e-aae932de61b9-audit-log\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.420247 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.420032    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhl5t\" (UniqueName: \"kubernetes.io/projected/79e7784b-6a83-4397-ad9e-aae932de61b9-kube-api-access-bhl5t\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.420247 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.420057    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/79e7784b-6a83-4397-ad9e-aae932de61b9-metrics-server-audit-profiles\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.420247 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.420084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/79e7784b-6a83-4397-ad9e-aae932de61b9-secret-metrics-server-tls\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.420247 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.420104    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/79e7784b-6a83-4397-ad9e-aae932de61b9-client-ca-bundle\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.420247 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.420155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/79e7784b-6a83-4397-ad9e-aae932de61b9-secret-metrics-server-client-certs\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.520918 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.520885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e7784b-6a83-4397-ad9e-aae932de61b9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.520918 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.520918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/79e7784b-6a83-4397-ad9e-aae932de61b9-audit-log\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.521196 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.520956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhl5t\" (UniqueName: \"kubernetes.io/projected/79e7784b-6a83-4397-ad9e-aae932de61b9-kube-api-access-bhl5t\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.521196 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.520987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/79e7784b-6a83-4397-ad9e-aae932de61b9-metrics-server-audit-profiles\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.521196 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.521012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/79e7784b-6a83-4397-ad9e-aae932de61b9-secret-metrics-server-tls\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.521196 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.521034    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/79e7784b-6a83-4397-ad9e-aae932de61b9-client-ca-bundle\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.521196 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.521174    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/79e7784b-6a83-4397-ad9e-aae932de61b9-secret-metrics-server-client-certs\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.521457 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.521433    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/79e7784b-6a83-4397-ad9e-aae932de61b9-audit-log\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.521725 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.521699    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e7784b-6a83-4397-ad9e-aae932de61b9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.522173 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.522146    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/79e7784b-6a83-4397-ad9e-aae932de61b9-metrics-server-audit-profiles\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.523731 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.523707    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/79e7784b-6a83-4397-ad9e-aae932de61b9-secret-metrics-server-tls\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.523731 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.523722    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/79e7784b-6a83-4397-ad9e-aae932de61b9-secret-metrics-server-client-certs\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.523731 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.523730    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/79e7784b-6a83-4397-ad9e-aae932de61b9-client-ca-bundle\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.529386 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.529367    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhl5t\" (UniqueName: \"kubernetes.io/projected/79e7784b-6a83-4397-ad9e-aae932de61b9-kube-api-access-bhl5t\") pod \"metrics-server-678f56c7df-qv25h\" (UID: \"79e7784b-6a83-4397-ad9e-aae932de61b9\") " pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.656036 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.655947    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:20.747312 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.747277    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:20.762292 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.762256    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj"]
Aug 04 18:01:20.765036 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.765013    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj"
Aug 04 18:01:20.767588 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.767568    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Aug 04 18:01:20.767711 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.767626    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gscvx\""
Aug 04 18:01:20.777194 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.777170    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj"]
Aug 04 18:01:20.795975 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.795951    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-678f56c7df-qv25h"]
Aug 04 18:01:20.798877 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:20.798855    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e7784b_6a83_4397_ad9e_aae932de61b9.slice/crio-9c9cfb4f56f32e5aedc888397d54a4178482738ef68a3ebe9d20a2b6f727f467 WatchSource:0}: Error finding container 9c9cfb4f56f32e5aedc888397d54a4178482738ef68a3ebe9d20a2b6f727f467: Status 404 returned error can't find the container with id 9c9cfb4f56f32e5aedc888397d54a4178482738ef68a3ebe9d20a2b6f727f467
Aug 04 18:01:20.823079 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.823052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxgjm\" (UniqueName: \"kubernetes.io/projected/7383bd99-ba08-47db-b119-17066b86f867-kube-api-access-mxgjm\") pod \"monitoring-plugin-6dffbbd67-hcjnj\" (UID: \"7383bd99-ba08-47db-b119-17066b86f867\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj"
Aug 04 18:01:20.823218 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.823151    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7383bd99-ba08-47db-b119-17066b86f867-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-hcjnj\" (UID: \"7383bd99-ba08-47db-b119-17066b86f867\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj"
Aug 04 18:01:20.844533 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.844504    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:20.905483 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.905440    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-678f56c7df-qv25h" event={"ID":"79e7784b-6a83-4397-ad9e-aae932de61b9","Type":"ContainerStarted","Data":"9c9cfb4f56f32e5aedc888397d54a4178482738ef68a3ebe9d20a2b6f727f467"}
Aug 04 18:01:20.924029 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.923953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7383bd99-ba08-47db-b119-17066b86f867-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-hcjnj\" (UID: \"7383bd99-ba08-47db-b119-17066b86f867\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj"
Aug 04 18:01:20.924029 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.923997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxgjm\" (UniqueName: \"kubernetes.io/projected/7383bd99-ba08-47db-b119-17066b86f867-kube-api-access-mxgjm\") pod \"monitoring-plugin-6dffbbd67-hcjnj\" (UID: \"7383bd99-ba08-47db-b119-17066b86f867\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj"
Aug 04 18:01:20.926762 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.926734    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7383bd99-ba08-47db-b119-17066b86f867-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-hcjnj\" (UID: \"7383bd99-ba08-47db-b119-17066b86f867\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj"
Aug 04 18:01:20.936347 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.936319    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dc4c4ccd9-kv9l7"]
Aug 04 18:01:20.938832 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.938805    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxgjm\" (UniqueName: \"kubernetes.io/projected/7383bd99-ba08-47db-b119-17066b86f867-kube-api-access-mxgjm\") pod \"monitoring-plugin-6dffbbd67-hcjnj\" (UID: \"7383bd99-ba08-47db-b119-17066b86f867\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj"
Aug 04 18:01:20.969264 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.969230    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c5786d48-k2qrh"]
Aug 04 18:01:20.972040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.971772    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:20.972040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.971970    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:20.982987 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:20.982961    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5786d48-k2qrh"]
Aug 04 18:01:21.025241 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.025206    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7da1159-037e-4155-8e4d-dab9a539e576-console-serving-cert\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.025423 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.025394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7da1159-037e-4155-8e4d-dab9a539e576-console-oauth-config\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.025507 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.025444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-oauth-serving-cert\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.025507 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.025486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-trusted-ca-bundle\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.025625 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.025521    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xwg\" (UniqueName: \"kubernetes.io/projected/b7da1159-037e-4155-8e4d-dab9a539e576-kube-api-access-n5xwg\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.025625 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.025603    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-service-ca\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.025731 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.025714    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-console-config\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.077165 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.077125    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj"
Aug 04 18:01:21.127099 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.127038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-console-config\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.127306 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.127140    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7da1159-037e-4155-8e4d-dab9a539e576-console-serving-cert\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.127306 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.127187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7da1159-037e-4155-8e4d-dab9a539e576-console-oauth-config\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.127306 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.127213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-oauth-serving-cert\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.127306 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.127250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-trusted-ca-bundle\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.127306 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.127284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xwg\" (UniqueName: \"kubernetes.io/projected/b7da1159-037e-4155-8e4d-dab9a539e576-kube-api-access-n5xwg\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.127547 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.127331    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-service-ca\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.127906 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.127875    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-console-config\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.128263 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.128194    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-trusted-ca-bundle\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.128263 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.128200    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-service-ca\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.128757 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.128498    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-oauth-serving-cert\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.130223 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.130199    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7da1159-037e-4155-8e4d-dab9a539e576-console-oauth-config\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.130310 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.130281    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7da1159-037e-4155-8e4d-dab9a539e576-console-serving-cert\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.141239 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.141211    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xwg\" (UniqueName: \"kubernetes.io/projected/b7da1159-037e-4155-8e4d-dab9a539e576-kube-api-access-n5xwg\") pod \"console-5c5786d48-k2qrh\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") " pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.241599 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.241570    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj"]
Aug 04 18:01:21.244386 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:21.244355    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7383bd99_ba08_47db_b119_17066b86f867.slice/crio-cbcd800609db0fab484ba2a7cd924fc5caf9f86ee169be61d138b0927fa7c799 WatchSource:0}: Error finding container cbcd800609db0fab484ba2a7cd924fc5caf9f86ee169be61d138b0927fa7c799: Status 404 returned error can't find the container with id cbcd800609db0fab484ba2a7cd924fc5caf9f86ee169be61d138b0927fa7c799
Aug 04 18:01:21.252832 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.252806    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"]
Aug 04 18:01:21.256352 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.256327    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.259803 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.259757    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Aug 04 18:01:21.259803 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.259789    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Aug 04 18:01:21.259955 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.259843    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Aug 04 18:01:21.259955 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.259889    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Aug 04 18:01:21.260445 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.260427    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lfkvp\""
Aug 04 18:01:21.260542 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.260446    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Aug 04 18:01:21.265302 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.265255    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Aug 04 18:01:21.280041 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.280009    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"]
Aug 04 18:01:21.284155 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.284131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:21.329071 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.329034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2492582-86ad-425c-9204-06faf9569f4b-serving-certs-ca-bundle\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.329255 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.329153    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2492582-86ad-425c-9204-06faf9569f4b-metrics-client-ca\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.329255 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.329202    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f2492582-86ad-425c-9204-06faf9569f4b-federate-client-tls\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.329255 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.329234    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f2492582-86ad-425c-9204-06faf9569f4b-secret-telemeter-client\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.329440 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.329279    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2492582-86ad-425c-9204-06faf9569f4b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.329440 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.329370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2492582-86ad-425c-9204-06faf9569f4b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.329440 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.329417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2fq6\" (UniqueName: \"kubernetes.io/projected/f2492582-86ad-425c-9204-06faf9569f4b-kube-api-access-b2fq6\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.329596 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.329519    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f2492582-86ad-425c-9204-06faf9569f4b-telemeter-client-tls\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.430337 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.430240    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2492582-86ad-425c-9204-06faf9569f4b-metrics-client-ca\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.430337 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.430303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f2492582-86ad-425c-9204-06faf9569f4b-federate-client-tls\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.430837 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.430335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f2492582-86ad-425c-9204-06faf9569f4b-secret-telemeter-client\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.430837 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.430379    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2492582-86ad-425c-9204-06faf9569f4b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.430837 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.430412    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2492582-86ad-425c-9204-06faf9569f4b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.430837 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.430441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2fq6\" (UniqueName: \"kubernetes.io/projected/f2492582-86ad-425c-9204-06faf9569f4b-kube-api-access-b2fq6\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.430837 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.430494    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f2492582-86ad-425c-9204-06faf9569f4b-telemeter-client-tls\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.430837 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.430542    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2492582-86ad-425c-9204-06faf9569f4b-serving-certs-ca-bundle\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.432315 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.431475    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2492582-86ad-425c-9204-06faf9569f4b-serving-certs-ca-bundle\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.432315 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.431924    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2492582-86ad-425c-9204-06faf9569f4b-metrics-client-ca\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.432315 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.432272    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2492582-86ad-425c-9204-06faf9569f4b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.434304 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.434280    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2492582-86ad-425c-9204-06faf9569f4b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.434860 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.434815    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f2492582-86ad-425c-9204-06faf9569f4b-secret-telemeter-client\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.435042 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.435020    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f2492582-86ad-425c-9204-06faf9569f4b-telemeter-client-tls\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.435447 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.435425    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f2492582-86ad-425c-9204-06faf9569f4b-federate-client-tls\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.440928 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.440881    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2fq6\" (UniqueName: \"kubernetes.io/projected/f2492582-86ad-425c-9204-06faf9569f4b-kube-api-access-b2fq6\") pod \"telemeter-client-5c5d56dc58-5h4n6\" (UID: \"f2492582-86ad-425c-9204-06faf9569f4b\") " pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.443384 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.443358    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5786d48-k2qrh"]
Aug 04 18:01:21.448578 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:21.448551    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7da1159_037e_4155_8e4d_dab9a539e576.slice/crio-fe920fc8a52094c6400a81a64e77cf1722d498e6c89c425811a465de27be8b70 WatchSource:0}: Error finding container fe920fc8a52094c6400a81a64e77cf1722d498e6c89c425811a465de27be8b70: Status 404 returned error can't find the container with id fe920fc8a52094c6400a81a64e77cf1722d498e6c89c425811a465de27be8b70
Aug 04 18:01:21.568640 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.568363    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"
Aug 04 18:01:21.761317 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.761196    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6"]
Aug 04 18:01:21.912898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.912860    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj" event={"ID":"7383bd99-ba08-47db-b119-17066b86f867","Type":"ContainerStarted","Data":"cbcd800609db0fab484ba2a7cd924fc5caf9f86ee169be61d138b0927fa7c799"}
Aug 04 18:01:21.914588 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.914527    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5786d48-k2qrh" event={"ID":"b7da1159-037e-4155-8e4d-dab9a539e576","Type":"ContainerStarted","Data":"bd8fd0a5611b48e4c4a9585a3898866b188d757a1f93afd38b5f4a82f03822be"}
Aug 04 18:01:21.914588 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.914574    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5786d48-k2qrh" event={"ID":"b7da1159-037e-4155-8e4d-dab9a539e576","Type":"ContainerStarted","Data":"fe920fc8a52094c6400a81a64e77cf1722d498e6c89c425811a465de27be8b70"}
Aug 04 18:01:21.943409 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:21.943293    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c5786d48-k2qrh" podStartSLOduration=1.943274423 podStartE2EDuration="1.943274423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:21.941682237 +0000 UTC m=+81.917513936" watchObservedRunningTime="2026-08-04 18:01:21.943274423 +0000 UTC m=+81.919106120"
Aug 04 18:01:22.246437 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.246341    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:01:22.249863 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.249834    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.251479 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.251395    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:22.252387 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.252363    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Aug 04 18:01:22.253150 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.253044    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-djou5n680khp2\""
Aug 04 18:01:22.253270 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.253230    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-brzkw\""
Aug 04 18:01:22.253351 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.253328    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Aug 04 18:01:22.253572 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.253549    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Aug 04 18:01:22.253953 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.253935    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Aug 04 18:01:22.254298 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.254279    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Aug 04 18:01:22.254386 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.254292    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Aug 04 18:01:22.255440 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.255283    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Aug 04 18:01:22.256849 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.256391    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Aug 04 18:01:22.257098 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.256958    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Aug 04 18:01:22.257740 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.257235    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Aug 04 18:01:22.257740 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.257425    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Aug 04 18:01:22.257740 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.257617    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Aug 04 18:01:22.258409 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.258034    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Aug 04 18:01:22.259179 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.259160    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Aug 04 18:01:22.273234 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.272778    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:01:22.300009 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:01:22.299970    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2492582_86ad_425c_9204_06faf9569f4b.slice/crio-a976f15f52b4c48facb165dc3ad58fe3c1bb24cdec828f96bdbed92ab5c50621 WatchSource:0}: Error finding container a976f15f52b4c48facb165dc3ad58fe3c1bb24cdec828f96bdbed92ab5c50621: Status 404 returned error can't find the container with id a976f15f52b4c48facb165dc3ad58fe3c1bb24cdec828f96bdbed92ab5c50621
Aug 04 18:01:22.345274 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345274 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345525 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345300    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345525 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345326    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345525 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345525 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345525 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345440    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345525 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345476    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hx7z\" (UniqueName: \"kubernetes.io/projected/a3385a98-db22-462f-9f7b-4e7178fd3ef0-kube-api-access-4hx7z\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345525 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345505    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3385a98-db22-462f-9f7b-4e7178fd3ef0-config-out\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345602    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3385a98-db22-462f-9f7b-4e7178fd3ef0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345803    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.345873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-web-config\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.346415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.345896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-config\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.447433 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.447390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.447878 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.447450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.447878 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.447480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.447878 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.447566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.447878 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.447622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.447878 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.447766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-web-config\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.447878 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.447823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-config\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.447878 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.447847    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.447878 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.447870    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.448287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.447899    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.448287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.447928    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.448287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.447965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.448287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.448005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.448287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.448029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.448287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.448068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hx7z\" (UniqueName: \"kubernetes.io/projected/a3385a98-db22-462f-9f7b-4e7178fd3ef0-kube-api-access-4hx7z\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.448287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.448099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.448287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.448155    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3385a98-db22-462f-9f7b-4e7178fd3ef0-config-out\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.448287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.448202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.448287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.448230    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.448287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.448259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3385a98-db22-462f-9f7b-4e7178fd3ef0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.449226 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.448846    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.449881 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.449484    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.452007 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.450419    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.452007 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.451012    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.452007 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.451297    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-config\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.452007 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.451419    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.452007 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.451859    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.452007 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.451898    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.452391 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.452206    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-web-config\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.452556 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.452495    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.453253 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.452791    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.453253 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.452899    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.453253 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.453197    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.453445 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.453305    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.454020 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.453996    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3385a98-db22-462f-9f7b-4e7178fd3ef0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.454351 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.454330    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3385a98-db22-462f-9f7b-4e7178fd3ef0-config-out\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.454501 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.454433    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.454591 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.454494    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.455563 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.455524    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.462378 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.462335    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hx7z\" (UniqueName: \"kubernetes.io/projected/a3385a98-db22-462f-9f7b-4e7178fd3ef0-kube-api-access-4hx7z\") pod \"prometheus-k8s-0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.563465 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.563432    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:22.820792 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.820734    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:22.821705 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.821681    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:22.831751 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.831504    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:22.924915 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.924883    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerStarted","Data":"c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5"}
Aug 04 18:01:22.927350 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.927263    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6" event={"ID":"f2492582-86ad-425c-9204-06faf9569f4b","Type":"ContainerStarted","Data":"a976f15f52b4c48facb165dc3ad58fe3c1bb24cdec828f96bdbed92ab5c50621"}
Aug 04 18:01:22.931246 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.930781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq" event={"ID":"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4","Type":"ContainerStarted","Data":"96638a905110680ff61ff6ad257e11d81e2f732949d35a5c3d0dad465e7425af"}
Aug 04 18:01:22.937062 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.936983    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:01:22.998535 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:22.998506    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:01:23.765686 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:23.765646    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:23.937755 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:23.937717    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerID="789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5" exitCode=0
Aug 04 18:01:23.937942 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:23.937799    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerDied","Data":"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"}
Aug 04 18:01:23.937942 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:23.937882    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerStarted","Data":"3302412aceb7f83fc26e44a87d8c0d54bb23e49a9859ffc230aebaca43a4c95b"}
Aug 04 18:01:23.940487 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:23.940461    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq" event={"ID":"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4","Type":"ContainerStarted","Data":"5cb4f32d28dfa0dddfd3118e805244adb5ecd822c67f83089746b60afe8a4c42"}
Aug 04 18:01:23.940587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:23.940529    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq" event={"ID":"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4","Type":"ContainerStarted","Data":"83ef894d7dcf3d226e46ed6de89ff1b2ce7d763777f12c0e4e423a035686d771"}
Aug 04 18:01:23.944446 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:23.944419    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerStarted","Data":"cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce"}
Aug 04 18:01:23.944567 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:23.944492    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerStarted","Data":"2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298"}
Aug 04 18:01:23.944567 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:23.944540    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerStarted","Data":"576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b"}
Aug 04 18:01:23.944679 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:23.944587    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerStarted","Data":"eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4"}
Aug 04 18:01:23.951401 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:23.951373    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:24.054990 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:24.054932    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:24.727313 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:24.727282    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:24.831467 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:24.831431    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:25.959185 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:25.959149    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6" event={"ID":"f2492582-86ad-425c-9204-06faf9569f4b","Type":"ContainerStarted","Data":"641e599681175ebadef9d6727dc31b528849d2b06505488da2c02e7398265d30"}
Aug 04 18:01:25.959665 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:25.959231    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6" event={"ID":"f2492582-86ad-425c-9204-06faf9569f4b","Type":"ContainerStarted","Data":"a8088370546c6c842f67377f237717758f30123e24818ab77d898967684780b5"}
Aug 04 18:01:25.959665 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:25.959269    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6" event={"ID":"f2492582-86ad-425c-9204-06faf9569f4b","Type":"ContainerStarted","Data":"cd48f0a609455f8ea0ac18c47f44e3ac8a04a942b57103619e418663e1637612"}
Aug 04 18:01:25.961003 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:25.960948    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-678f56c7df-qv25h" event={"ID":"79e7784b-6a83-4397-ad9e-aae932de61b9","Type":"ContainerStarted","Data":"861754c79620847c183c218d95db21a461a010ab0f9c60a4de8c7c55641bcfb5"}
Aug 04 18:01:25.964477 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:25.964446    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq" event={"ID":"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4","Type":"ContainerStarted","Data":"35c124ac0ab729250d4fbe51238e8d69b6173ff2c7996f19b3897dafad85bfa9"}
Aug 04 18:01:25.964592 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:25.964517    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq" event={"ID":"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4","Type":"ContainerStarted","Data":"fe04429b8e615ed28cf15652ce9abe91c455ff287981000b8a43017008d0e2e1"}
Aug 04 18:01:25.964592 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:25.964563    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq" event={"ID":"8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4","Type":"ContainerStarted","Data":"f0a316f0046fae7c5b57f58f0510c8ead9d44690ef7f3910729b253ea1ae4b12"}
Aug 04 18:01:25.964698 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:25.964610    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:25.966198 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:25.966170    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj" event={"ID":"7383bd99-ba08-47db-b119-17066b86f867","Type":"ContainerStarted","Data":"ecd654186bcb6a463691f54e71ced836864f566b11257636ed2ce0d152ec2ac4"}
Aug 04 18:01:25.966355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:25.966332    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj"
Aug 04 18:01:25.970716 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:25.970628    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerStarted","Data":"91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c"}
Aug 04 18:01:25.972237 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:25.972216    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj"
Aug 04 18:01:25.989769 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:25.989716    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5c5d56dc58-5h4n6" podStartSLOduration=2.431061601 podStartE2EDuration="4.98970072s" totalImagesPullingTime="2.558639119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:25.986725996 +0000 UTC m=+85.962557700" watchObservedRunningTime="2026-08-04 18:01:25.98970072 +0000 UTC m=+85.965532417"
Aug 04 18:01:26.007926 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:26.007870    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-678f56c7df-qv25h" podStartSLOduration=1.943391562 podStartE2EDuration="6.007852613s" totalImagesPullingTime="4.064461051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:26.006342102 +0000 UTC m=+85.982173838" watchObservedRunningTime="2026-08-04 18:01:26.007852613 +0000 UTC m=+85.983684310"
Aug 04 18:01:26.036610 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:26.036558    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq" podStartSLOduration=1.939298588 podStartE2EDuration="7.036543111s" totalImagesPullingTime="5.097244523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:26.033326388 +0000 UTC m=+86.009158092" watchObservedRunningTime="2026-08-04 18:01:26.036543111 +0000 UTC m=+86.012374808"
Aug 04 18:01:26.066083 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:26.066015    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.329358031 podStartE2EDuration="9.065989976s" totalImagesPullingTime="5.714523253s" totalInitContainerRuntime="22.108692ms" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:26.064786374 +0000 UTC m=+86.040618079" watchObservedRunningTime="2026-08-04 18:01:26.065989976 +0000 UTC m=+86.041821670"
Aug 04 18:01:26.097180 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:26.097130    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hcjnj" podStartSLOduration=2.478419891 podStartE2EDuration="6.097098324s" totalImagesPullingTime="3.618678433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:26.094567093 +0000 UTC m=+86.070398796" watchObservedRunningTime="2026-08-04 18:01:26.097098324 +0000 UTC m=+86.072930020"
Aug 04 18:01:26.147986 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:26.147949    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:26.248867 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:26.248782    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:26.350923 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:26.350886    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:27.528479 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:27.528438    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c5786d48-k2qrh"]
Aug 04 18:01:27.982941 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:27.982905    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerStarted","Data":"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"}
Aug 04 18:01:27.983086 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:27.982978    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerStarted","Data":"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"}
Aug 04 18:01:27.983086 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:27.983006    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerStarted","Data":"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"}
Aug 04 18:01:27.983086 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:27.983032    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerStarted","Data":"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"}
Aug 04 18:01:27.983086 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:27.983057    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerStarted","Data":"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"}
Aug 04 18:01:28.991084 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:28.991050    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerStarted","Data":"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"}
Aug 04 18:01:29.031628 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:29.031564    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.366560488 podStartE2EDuration="7.031543073s" totalImagesPullingTime="3.634189458s" totalInitContainerRuntime="30.793127ms" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:29.028821161 +0000 UTC m=+89.004652870" watchObservedRunningTime="2026-08-04 18:01:29.031543073 +0000 UTC m=+89.007374771"
Aug 04 18:01:31.285178 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:31.285133    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:31.980942 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:31.980906    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c679f8c78-2rshq"
Aug 04 18:01:32.563769 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:32.563720    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:32.919887 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:32.919806    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:32.987227 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:32.987186    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:38.727263 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:38.727215    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d55549997-cj5gb"]
Aug 04 18:01:38.819554 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:38.819514    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:38.878619 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:38.878586    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:38.941606 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:38.941568    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:40.656806 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:40.656770    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:40.657208 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:40.656833    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:01:44.861449 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:44.861416    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q5kpw"
Aug 04 18:01:45.957516 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:45.957457    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6dc4c4ccd9-kv9l7" podUID="716abfd2-ee03-4ebc-bb89-d41d82d26090" containerName="console" containerID="cri-o://b28221a866ff55b85444968db2483f55c114b83771f5239b586d9249ade1f368" gracePeriod=15
Aug 04 18:01:46.201272 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.201239    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:46.380213 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.380175    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-oauth-config\") pod \"716abfd2-ee03-4ebc-bb89-d41d82d26090\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") "
Aug 04 18:01:46.380388 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.380226    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-oauth-serving-cert\") pod \"716abfd2-ee03-4ebc-bb89-d41d82d26090\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") "
Aug 04 18:01:46.380388 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.380255    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-service-ca\") pod \"716abfd2-ee03-4ebc-bb89-d41d82d26090\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") "
Aug 04 18:01:46.380388 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.380332    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-serving-cert\") pod \"716abfd2-ee03-4ebc-bb89-d41d82d26090\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") "
Aug 04 18:01:46.380388 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.380356    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-config\") pod \"716abfd2-ee03-4ebc-bb89-d41d82d26090\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") "
Aug 04 18:01:46.380388 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.380372    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jwfd\" (UniqueName: \"kubernetes.io/projected/716abfd2-ee03-4ebc-bb89-d41d82d26090-kube-api-access-5jwfd\") pod \"716abfd2-ee03-4ebc-bb89-d41d82d26090\" (UID: \"716abfd2-ee03-4ebc-bb89-d41d82d26090\") "
Aug 04 18:01:46.380812 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.380783    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-service-ca" (OuterVolumeSpecName: "service-ca") pod "716abfd2-ee03-4ebc-bb89-d41d82d26090" (UID: "716abfd2-ee03-4ebc-bb89-d41d82d26090"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:46.380887 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.380710    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-config" (OuterVolumeSpecName: "console-config") pod "716abfd2-ee03-4ebc-bb89-d41d82d26090" (UID: "716abfd2-ee03-4ebc-bb89-d41d82d26090"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:46.380927 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.380877    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "716abfd2-ee03-4ebc-bb89-d41d82d26090" (UID: "716abfd2-ee03-4ebc-bb89-d41d82d26090"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:46.382755 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.382734    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "716abfd2-ee03-4ebc-bb89-d41d82d26090" (UID: "716abfd2-ee03-4ebc-bb89-d41d82d26090"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:01:46.382939 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.382918    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716abfd2-ee03-4ebc-bb89-d41d82d26090-kube-api-access-5jwfd" (OuterVolumeSpecName: "kube-api-access-5jwfd") pod "716abfd2-ee03-4ebc-bb89-d41d82d26090" (UID: "716abfd2-ee03-4ebc-bb89-d41d82d26090"). InnerVolumeSpecName "kube-api-access-5jwfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:01:46.382999 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.382920    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "716abfd2-ee03-4ebc-bb89-d41d82d26090" (UID: "716abfd2-ee03-4ebc-bb89-d41d82d26090"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:01:46.481550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.481514    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-serving-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:01:46.481550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.481546    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:01:46.481550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.481559    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jwfd\" (UniqueName: \"kubernetes.io/projected/716abfd2-ee03-4ebc-bb89-d41d82d26090-kube-api-access-5jwfd\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:01:46.481776 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.481571    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/716abfd2-ee03-4ebc-bb89-d41d82d26090-console-oauth-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:01:46.481776 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.481580    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-oauth-serving-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:01:46.481776 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:46.481589    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/716abfd2-ee03-4ebc-bb89-d41d82d26090-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:01:47.075524 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:47.075492    2576 generic.go:356] "Generic (PLEG): container finished" podID="716abfd2-ee03-4ebc-bb89-d41d82d26090" containerID="b28221a866ff55b85444968db2483f55c114b83771f5239b586d9249ade1f368" exitCode=0
Aug 04 18:01:47.075946 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:47.075538    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc4c4ccd9-kv9l7"
Aug 04 18:01:47.075946 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:47.075538    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc4c4ccd9-kv9l7" event={"ID":"716abfd2-ee03-4ebc-bb89-d41d82d26090","Type":"ContainerDied","Data":"b28221a866ff55b85444968db2483f55c114b83771f5239b586d9249ade1f368"}
Aug 04 18:01:47.075946 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:47.075667    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc4c4ccd9-kv9l7" event={"ID":"716abfd2-ee03-4ebc-bb89-d41d82d26090","Type":"ContainerDied","Data":"d53ac1d0383c895572a9cdfb58ca82426328890cc130cba5bc18eb47615207d6"}
Aug 04 18:01:47.075946 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:47.075706    2576 scope.go:117] "RemoveContainer" containerID="b28221a866ff55b85444968db2483f55c114b83771f5239b586d9249ade1f368"
Aug 04 18:01:47.085251 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:47.085233    2576 scope.go:117] "RemoveContainer" containerID="b28221a866ff55b85444968db2483f55c114b83771f5239b586d9249ade1f368"
Aug 04 18:01:47.085536 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:01:47.085517    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28221a866ff55b85444968db2483f55c114b83771f5239b586d9249ade1f368\": container with ID starting with b28221a866ff55b85444968db2483f55c114b83771f5239b586d9249ade1f368 not found: ID does not exist" containerID="b28221a866ff55b85444968db2483f55c114b83771f5239b586d9249ade1f368"
Aug 04 18:01:47.085606 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:47.085543    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28221a866ff55b85444968db2483f55c114b83771f5239b586d9249ade1f368"} err="failed to get container status \"b28221a866ff55b85444968db2483f55c114b83771f5239b586d9249ade1f368\": rpc error: code = NotFound desc = could not find container \"b28221a866ff55b85444968db2483f55c114b83771f5239b586d9249ade1f368\": container with ID starting with b28221a866ff55b85444968db2483f55c114b83771f5239b586d9249ade1f368 not found: ID does not exist"
Aug 04 18:01:47.094163 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:47.094140    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dc4c4ccd9-kv9l7"]
Aug 04 18:01:47.099315 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:47.099292    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dc4c4ccd9-kv9l7"]
Aug 04 18:01:47.930292 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:47.930239    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:47.990120 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:47.990064    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:48.581412 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:48.581370    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="716abfd2-ee03-4ebc-bb89-d41d82d26090" path="/var/lib/kubelet/pods/716abfd2-ee03-4ebc-bb89-d41d82d26090/volumes"
Aug 04 18:01:50.948771 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:50.948728    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:51.022072 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:51.022035    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:52.550481 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.550437    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c5786d48-k2qrh" podUID="b7da1159-037e-4155-8e4d-dab9a539e576" containerName="console" containerID="cri-o://bd8fd0a5611b48e4c4a9585a3898866b188d757a1f93afd38b5f4a82f03822be" gracePeriod=15
Aug 04 18:01:52.791538 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.791510    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:52.838504 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.838414    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-console-config\") pod \"b7da1159-037e-4155-8e4d-dab9a539e576\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") "
Aug 04 18:01:52.838504 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.838454    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-trusted-ca-bundle\") pod \"b7da1159-037e-4155-8e4d-dab9a539e576\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") "
Aug 04 18:01:52.838690 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.838515    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7da1159-037e-4155-8e4d-dab9a539e576-console-oauth-config\") pod \"b7da1159-037e-4155-8e4d-dab9a539e576\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") "
Aug 04 18:01:52.838690 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.838582    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5xwg\" (UniqueName: \"kubernetes.io/projected/b7da1159-037e-4155-8e4d-dab9a539e576-kube-api-access-n5xwg\") pod \"b7da1159-037e-4155-8e4d-dab9a539e576\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") "
Aug 04 18:01:52.838690 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.838636    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-oauth-serving-cert\") pod \"b7da1159-037e-4155-8e4d-dab9a539e576\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") "
Aug 04 18:01:52.838690 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.838670    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7da1159-037e-4155-8e4d-dab9a539e576-console-serving-cert\") pod \"b7da1159-037e-4155-8e4d-dab9a539e576\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") "
Aug 04 18:01:52.838690 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.838691    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-service-ca\") pod \"b7da1159-037e-4155-8e4d-dab9a539e576\" (UID: \"b7da1159-037e-4155-8e4d-dab9a539e576\") "
Aug 04 18:01:52.838978 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.838835    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-console-config" (OuterVolumeSpecName: "console-config") pod "b7da1159-037e-4155-8e4d-dab9a539e576" (UID: "b7da1159-037e-4155-8e4d-dab9a539e576"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:52.839141 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.838985    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b7da1159-037e-4155-8e4d-dab9a539e576" (UID: "b7da1159-037e-4155-8e4d-dab9a539e576"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:52.839141 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.839023    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b7da1159-037e-4155-8e4d-dab9a539e576" (UID: "b7da1159-037e-4155-8e4d-dab9a539e576"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:52.839250 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.839101    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-service-ca" (OuterVolumeSpecName: "service-ca") pod "b7da1159-037e-4155-8e4d-dab9a539e576" (UID: "b7da1159-037e-4155-8e4d-dab9a539e576"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:52.840762 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.840737    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7da1159-037e-4155-8e4d-dab9a539e576-kube-api-access-n5xwg" (OuterVolumeSpecName: "kube-api-access-n5xwg") pod "b7da1159-037e-4155-8e4d-dab9a539e576" (UID: "b7da1159-037e-4155-8e4d-dab9a539e576"). InnerVolumeSpecName "kube-api-access-n5xwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:01:52.840762 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.840755    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7da1159-037e-4155-8e4d-dab9a539e576-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b7da1159-037e-4155-8e4d-dab9a539e576" (UID: "b7da1159-037e-4155-8e4d-dab9a539e576"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:01:52.840873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.840862    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7da1159-037e-4155-8e4d-dab9a539e576-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b7da1159-037e-4155-8e4d-dab9a539e576" (UID: "b7da1159-037e-4155-8e4d-dab9a539e576"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:01:52.939627 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.939596    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7da1159-037e-4155-8e4d-dab9a539e576-console-oauth-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:01:52.939627 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.939626    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5xwg\" (UniqueName: \"kubernetes.io/projected/b7da1159-037e-4155-8e4d-dab9a539e576-kube-api-access-n5xwg\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:01:52.939833 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.939639    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-oauth-serving-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:01:52.939833 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.939651    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7da1159-037e-4155-8e4d-dab9a539e576-console-serving-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:01:52.939833 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.939663    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:01:52.939833 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.939674    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-console-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:01:52.939833 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:52.939684    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7da1159-037e-4155-8e4d-dab9a539e576-trusted-ca-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:01:53.102442 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:53.102347    2576 generic.go:356] "Generic (PLEG): container finished" podID="b7da1159-037e-4155-8e4d-dab9a539e576" containerID="bd8fd0a5611b48e4c4a9585a3898866b188d757a1f93afd38b5f4a82f03822be" exitCode=0
Aug 04 18:01:53.102442 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:53.102393    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5786d48-k2qrh"
Aug 04 18:01:53.102672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:53.102437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5786d48-k2qrh" event={"ID":"b7da1159-037e-4155-8e4d-dab9a539e576","Type":"ContainerDied","Data":"bd8fd0a5611b48e4c4a9585a3898866b188d757a1f93afd38b5f4a82f03822be"}
Aug 04 18:01:53.102672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:53.102491    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5786d48-k2qrh" event={"ID":"b7da1159-037e-4155-8e4d-dab9a539e576","Type":"ContainerDied","Data":"fe920fc8a52094c6400a81a64e77cf1722d498e6c89c425811a465de27be8b70"}
Aug 04 18:01:53.102672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:53.102516    2576 scope.go:117] "RemoveContainer" containerID="bd8fd0a5611b48e4c4a9585a3898866b188d757a1f93afd38b5f4a82f03822be"
Aug 04 18:01:53.110970 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:53.110954    2576 scope.go:117] "RemoveContainer" containerID="bd8fd0a5611b48e4c4a9585a3898866b188d757a1f93afd38b5f4a82f03822be"
Aug 04 18:01:53.111280 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:01:53.111259    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8fd0a5611b48e4c4a9585a3898866b188d757a1f93afd38b5f4a82f03822be\": container with ID starting with bd8fd0a5611b48e4c4a9585a3898866b188d757a1f93afd38b5f4a82f03822be not found: ID does not exist" containerID="bd8fd0a5611b48e4c4a9585a3898866b188d757a1f93afd38b5f4a82f03822be"
Aug 04 18:01:53.111349 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:53.111291    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8fd0a5611b48e4c4a9585a3898866b188d757a1f93afd38b5f4a82f03822be"} err="failed to get container status \"bd8fd0a5611b48e4c4a9585a3898866b188d757a1f93afd38b5f4a82f03822be\": rpc error: code = NotFound desc = could not find container \"bd8fd0a5611b48e4c4a9585a3898866b188d757a1f93afd38b5f4a82f03822be\": container with ID starting with bd8fd0a5611b48e4c4a9585a3898866b188d757a1f93afd38b5f4a82f03822be not found: ID does not exist"
Aug 04 18:01:53.125526 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:53.125499    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c5786d48-k2qrh"]
Aug 04 18:01:53.131358 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:53.131331    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c5786d48-k2qrh"]
Aug 04 18:01:54.580945 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:01:54.580912    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7da1159-037e-4155-8e4d-dab9a539e576" path="/var/lib/kubelet/pods/b7da1159-037e-4155-8e4d-dab9a539e576/volumes"
Aug 04 18:02:00.661773 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:00.661742    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:02:00.665525 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:00.665458    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-678f56c7df-qv25h"
Aug 04 18:02:01.372204 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:01.372143    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:03.746095 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:03.746037    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d55549997-cj5gb" podUID="4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623" containerName="console" containerID="cri-o://3d22bf370ea8a798beb5673921a6cb01039feb5723d0f2e28e624306811abd0f" gracePeriod=15
Aug 04 18:02:03.989770 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:03.989742    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:02:04.044795 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.044769    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-oauth-serving-cert\") pod \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") "
Aug 04 18:02:04.044935 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.044808    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhm4v\" (UniqueName: \"kubernetes.io/projected/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-kube-api-access-hhm4v\") pod \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") "
Aug 04 18:02:04.044935 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.044847    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-service-ca\") pod \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") "
Aug 04 18:02:04.044935 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.044883    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-serving-cert\") pod \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") "
Aug 04 18:02:04.044935 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.044920    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-trusted-ca-bundle\") pod \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") "
Aug 04 18:02:04.045140 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.044945    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-oauth-config\") pod \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") "
Aug 04 18:02:04.045140 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.045086    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-config\") pod \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\" (UID: \"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623\") "
Aug 04 18:02:04.045347 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.045318    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623" (UID: "4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:04.045347 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.045319    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-service-ca" (OuterVolumeSpecName: "service-ca") pod "4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623" (UID: "4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:04.045521 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.045415    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:04.045521 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.045436    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-oauth-serving-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:04.045521 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.045497    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623" (UID: "4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:04.045634 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.045555    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-config" (OuterVolumeSpecName: "console-config") pod "4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623" (UID: "4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:04.047140 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.047071    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-kube-api-access-hhm4v" (OuterVolumeSpecName: "kube-api-access-hhm4v") pod "4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623" (UID: "4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623"). InnerVolumeSpecName "kube-api-access-hhm4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:04.047243 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.047215    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623" (UID: "4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:04.047327 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.047304    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623" (UID: "4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:04.146062 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.146028    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhm4v\" (UniqueName: \"kubernetes.io/projected/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-kube-api-access-hhm4v\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:04.146062 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.146053    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-serving-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:04.146062 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.146062    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-trusted-ca-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:04.146062 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.146071    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-oauth-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:04.146337 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.146081    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623-console-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:04.148158 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.148132    2576 generic.go:356] "Generic (PLEG): container finished" podID="4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623" containerID="3d22bf370ea8a798beb5673921a6cb01039feb5723d0f2e28e624306811abd0f" exitCode=0
Aug 04 18:02:04.148234 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.148170    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d55549997-cj5gb"
Aug 04 18:02:04.148234 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.148174    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d55549997-cj5gb" event={"ID":"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623","Type":"ContainerDied","Data":"3d22bf370ea8a798beb5673921a6cb01039feb5723d0f2e28e624306811abd0f"}
Aug 04 18:02:04.148294 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.148247    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d55549997-cj5gb" event={"ID":"4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623","Type":"ContainerDied","Data":"8bd62201952cf32eda23eacccb22a3fceface268b95aa4b301a35a5a901a68c1"}
Aug 04 18:02:04.148294 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.148269    2576 scope.go:117] "RemoveContainer" containerID="3d22bf370ea8a798beb5673921a6cb01039feb5723d0f2e28e624306811abd0f"
Aug 04 18:02:04.156838 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.156820    2576 scope.go:117] "RemoveContainer" containerID="3d22bf370ea8a798beb5673921a6cb01039feb5723d0f2e28e624306811abd0f"
Aug 04 18:02:04.157084 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:04.157066    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d22bf370ea8a798beb5673921a6cb01039feb5723d0f2e28e624306811abd0f\": container with ID starting with 3d22bf370ea8a798beb5673921a6cb01039feb5723d0f2e28e624306811abd0f not found: ID does not exist" containerID="3d22bf370ea8a798beb5673921a6cb01039feb5723d0f2e28e624306811abd0f"
Aug 04 18:02:04.157154 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.157094    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d22bf370ea8a798beb5673921a6cb01039feb5723d0f2e28e624306811abd0f"} err="failed to get container status \"3d22bf370ea8a798beb5673921a6cb01039feb5723d0f2e28e624306811abd0f\": rpc error: code = NotFound desc = could not find container \"3d22bf370ea8a798beb5673921a6cb01039feb5723d0f2e28e624306811abd0f\": container with ID starting with 3d22bf370ea8a798beb5673921a6cb01039feb5723d0f2e28e624306811abd0f not found: ID does not exist"
Aug 04 18:02:04.171441 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.171398    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d55549997-cj5gb"]
Aug 04 18:02:04.179179 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.179150    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d55549997-cj5gb"]
Aug 04 18:02:04.580828 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:04.580795    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623" path="/var/lib/kubelet/pods/4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623/volumes"
Aug 04 18:02:22.564182 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:22.564141    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:22.583771 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:22.583747    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:23.237473 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:23.237448    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:32.111544 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:32.111504    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:32.179802 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:32.179772    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:34.179328 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:34.179287    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:36.459672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:36.459635    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:02:36.460143 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:36.459943    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="alertmanager" containerID="cri-o://c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5" gracePeriod=120
Aug 04 18:02:36.460143 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:36.460009    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="kube-rbac-proxy-metric" containerID="cri-o://cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce" gracePeriod=120
Aug 04 18:02:36.460143 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:36.460047    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="prom-label-proxy" containerID="cri-o://91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c" gracePeriod=120
Aug 04 18:02:36.460143 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:36.460029    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="kube-rbac-proxy" containerID="cri-o://2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298" gracePeriod=120
Aug 04 18:02:36.460143 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:36.460061    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="config-reloader" containerID="cri-o://eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4" gracePeriod=120
Aug 04 18:02:36.460143 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:36.460006    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="kube-rbac-proxy-web" containerID="cri-o://576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b" gracePeriod=120
Aug 04 18:02:37.283636 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.283597    2576 generic.go:356] "Generic (PLEG): container finished" podID="9aa34905-832c-4e14-a6b0-77304fc88782" containerID="91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c" exitCode=0
Aug 04 18:02:37.283636 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.283624    2576 generic.go:356] "Generic (PLEG): container finished" podID="9aa34905-832c-4e14-a6b0-77304fc88782" containerID="2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298" exitCode=0
Aug 04 18:02:37.283636 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.283633    2576 generic.go:356] "Generic (PLEG): container finished" podID="9aa34905-832c-4e14-a6b0-77304fc88782" containerID="eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4" exitCode=0
Aug 04 18:02:37.283636 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.283640    2576 generic.go:356] "Generic (PLEG): container finished" podID="9aa34905-832c-4e14-a6b0-77304fc88782" containerID="c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5" exitCode=0
Aug 04 18:02:37.283908 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.283664    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerDied","Data":"91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c"}
Aug 04 18:02:37.283908 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.283724    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerDied","Data":"2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298"}
Aug 04 18:02:37.283908 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.283750    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerDied","Data":"eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4"}
Aug 04 18:02:37.283908 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.283775    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerDied","Data":"c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5"}
Aug 04 18:02:37.705328 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.705304    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:37.750449 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.750422    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa34905-832c-4e14-a6b0-77304fc88782-tls-assets\") pod \"9aa34905-832c-4e14-a6b0-77304fc88782\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") "
Aug 04 18:02:37.750616 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.750475    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy-web\") pod \"9aa34905-832c-4e14-a6b0-77304fc88782\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") "
Aug 04 18:02:37.750616 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.750508    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy\") pod \"9aa34905-832c-4e14-a6b0-77304fc88782\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") "
Aug 04 18:02:37.750616 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.750525    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-config-volume\") pod \"9aa34905-832c-4e14-a6b0-77304fc88782\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") "
Aug 04 18:02:37.750616 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.750556    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-web-config\") pod \"9aa34905-832c-4e14-a6b0-77304fc88782\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") "
Aug 04 18:02:37.750616 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.750591    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-cluster-tls-config\") pod \"9aa34905-832c-4e14-a6b0-77304fc88782\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") "
Aug 04 18:02:37.750883 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.750619    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa34905-832c-4e14-a6b0-77304fc88782-metrics-client-ca\") pod \"9aa34905-832c-4e14-a6b0-77304fc88782\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") "
Aug 04 18:02:37.750883 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.750644    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aa34905-832c-4e14-a6b0-77304fc88782-alertmanager-trusted-ca-bundle\") pod \"9aa34905-832c-4e14-a6b0-77304fc88782\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") "
Aug 04 18:02:37.750883 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.750680    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa34905-832c-4e14-a6b0-77304fc88782-alertmanager-main-db\") pod \"9aa34905-832c-4e14-a6b0-77304fc88782\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") "
Aug 04 18:02:37.750883 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.750705    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-main-tls\") pod \"9aa34905-832c-4e14-a6b0-77304fc88782\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") "
Aug 04 18:02:37.750883 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.750746    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s8z5\" (UniqueName: \"kubernetes.io/projected/9aa34905-832c-4e14-a6b0-77304fc88782-kube-api-access-5s8z5\") pod \"9aa34905-832c-4e14-a6b0-77304fc88782\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") "
Aug 04 18:02:37.750883 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.750779    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9aa34905-832c-4e14-a6b0-77304fc88782\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") "
Aug 04 18:02:37.750883 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.750808    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa34905-832c-4e14-a6b0-77304fc88782-config-out\") pod \"9aa34905-832c-4e14-a6b0-77304fc88782\" (UID: \"9aa34905-832c-4e14-a6b0-77304fc88782\") "
Aug 04 18:02:37.751249 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.751224    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa34905-832c-4e14-a6b0-77304fc88782-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "9aa34905-832c-4e14-a6b0-77304fc88782" (UID: "9aa34905-832c-4e14-a6b0-77304fc88782"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:37.751650 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.751620    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa34905-832c-4e14-a6b0-77304fc88782-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "9aa34905-832c-4e14-a6b0-77304fc88782" (UID: "9aa34905-832c-4e14-a6b0-77304fc88782"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:37.751940 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.751915    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa34905-832c-4e14-a6b0-77304fc88782-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "9aa34905-832c-4e14-a6b0-77304fc88782" (UID: "9aa34905-832c-4e14-a6b0-77304fc88782"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:02:37.754135 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.754093    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "9aa34905-832c-4e14-a6b0-77304fc88782" (UID: "9aa34905-832c-4e14-a6b0-77304fc88782"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:37.754260 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.754150    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa34905-832c-4e14-a6b0-77304fc88782-config-out" (OuterVolumeSpecName: "config-out") pod "9aa34905-832c-4e14-a6b0-77304fc88782" (UID: "9aa34905-832c-4e14-a6b0-77304fc88782"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:02:37.754497 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.754451    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "9aa34905-832c-4e14-a6b0-77304fc88782" (UID: "9aa34905-832c-4e14-a6b0-77304fc88782"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:37.756058 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.756024    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "9aa34905-832c-4e14-a6b0-77304fc88782" (UID: "9aa34905-832c-4e14-a6b0-77304fc88782"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:37.756229 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.756202    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "9aa34905-832c-4e14-a6b0-77304fc88782" (UID: "9aa34905-832c-4e14-a6b0-77304fc88782"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:37.756329 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.756304    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-config-volume" (OuterVolumeSpecName: "config-volume") pod "9aa34905-832c-4e14-a6b0-77304fc88782" (UID: "9aa34905-832c-4e14-a6b0-77304fc88782"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:37.756389 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.756321    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa34905-832c-4e14-a6b0-77304fc88782-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9aa34905-832c-4e14-a6b0-77304fc88782" (UID: "9aa34905-832c-4e14-a6b0-77304fc88782"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:37.756495 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.756474    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa34905-832c-4e14-a6b0-77304fc88782-kube-api-access-5s8z5" (OuterVolumeSpecName: "kube-api-access-5s8z5") pod "9aa34905-832c-4e14-a6b0-77304fc88782" (UID: "9aa34905-832c-4e14-a6b0-77304fc88782"). InnerVolumeSpecName "kube-api-access-5s8z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:37.761520 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.761403    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "9aa34905-832c-4e14-a6b0-77304fc88782" (UID: "9aa34905-832c-4e14-a6b0-77304fc88782"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:37.767603 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.767582    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-web-config" (OuterVolumeSpecName: "web-config") pod "9aa34905-832c-4e14-a6b0-77304fc88782" (UID: "9aa34905-832c-4e14-a6b0-77304fc88782"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:37.852412 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.852371    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:37.852412 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.852407    2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-config-volume\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:37.852412 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.852422    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-web-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:37.852676 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.852435    2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-cluster-tls-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:37.852676 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.852447    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa34905-832c-4e14-a6b0-77304fc88782-metrics-client-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:37.852676 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.852458    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aa34905-832c-4e14-a6b0-77304fc88782-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:37.852676 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.852469    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa34905-832c-4e14-a6b0-77304fc88782-alertmanager-main-db\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:37.852676 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.852480    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-main-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:37.852676 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.852493    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s8z5\" (UniqueName: \"kubernetes.io/projected/9aa34905-832c-4e14-a6b0-77304fc88782-kube-api-access-5s8z5\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:37.852676 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.852506    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:37.852676 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.852520    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa34905-832c-4e14-a6b0-77304fc88782-config-out\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:37.852676 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.852530    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa34905-832c-4e14-a6b0-77304fc88782-tls-assets\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:37.852676 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:37.852541    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aa34905-832c-4e14-a6b0-77304fc88782-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:38.290653 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.290619    2576 generic.go:356] "Generic (PLEG): container finished" podID="9aa34905-832c-4e14-a6b0-77304fc88782" containerID="cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce" exitCode=0
Aug 04 18:02:38.290653 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.290645    2576 generic.go:356] "Generic (PLEG): container finished" podID="9aa34905-832c-4e14-a6b0-77304fc88782" containerID="576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b" exitCode=0
Aug 04 18:02:38.290875 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.290701    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.290875 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.290694    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerDied","Data":"cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce"}
Aug 04 18:02:38.290875 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.290835    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerDied","Data":"576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b"}
Aug 04 18:02:38.290875 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.290867    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa34905-832c-4e14-a6b0-77304fc88782","Type":"ContainerDied","Data":"248c563f97077ae6bbdd7724e3b3f5f91b2823c469ff302e4b2345eaf27c50d1"}
Aug 04 18:02:38.291006 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.290897    2576 scope.go:117] "RemoveContainer" containerID="91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c"
Aug 04 18:02:38.300211 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.300190    2576 scope.go:117] "RemoveContainer" containerID="cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce"
Aug 04 18:02:38.307505 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.307489    2576 scope.go:117] "RemoveContainer" containerID="2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298"
Aug 04 18:02:38.314297 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.314269    2576 scope.go:117] "RemoveContainer" containerID="576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b"
Aug 04 18:02:38.316856 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.316832    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:02:38.321520 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.321495    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:02:38.322520 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.322502    2576 scope.go:117] "RemoveContainer" containerID="eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4"
Aug 04 18:02:38.329782 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.329764    2576 scope.go:117] "RemoveContainer" containerID="c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5"
Aug 04 18:02:38.336832 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.336814    2576 scope.go:117] "RemoveContainer" containerID="504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e"
Aug 04 18:02:38.344312 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.344286    2576 scope.go:117] "RemoveContainer" containerID="91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c"
Aug 04 18:02:38.344579 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:38.344560    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c\": container with ID starting with 91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c not found: ID does not exist" containerID="91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c"
Aug 04 18:02:38.344653 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.344594    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c"} err="failed to get container status \"91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c\": rpc error: code = NotFound desc = could not find container \"91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c\": container with ID starting with 91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c not found: ID does not exist"
Aug 04 18:02:38.344653 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.344621    2576 scope.go:117] "RemoveContainer" containerID="cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce"
Aug 04 18:02:38.344874 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:38.344858    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce\": container with ID starting with cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce not found: ID does not exist" containerID="cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce"
Aug 04 18:02:38.344925 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.344880    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce"} err="failed to get container status \"cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce\": rpc error: code = NotFound desc = could not find container \"cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce\": container with ID starting with cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce not found: ID does not exist"
Aug 04 18:02:38.344925 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.344897    2576 scope.go:117] "RemoveContainer" containerID="2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298"
Aug 04 18:02:38.345104 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:38.345088    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298\": container with ID starting with 2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298 not found: ID does not exist" containerID="2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298"
Aug 04 18:02:38.345182 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.345127    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298"} err="failed to get container status \"2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298\": rpc error: code = NotFound desc = could not find container \"2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298\": container with ID starting with 2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298 not found: ID does not exist"
Aug 04 18:02:38.345182 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.345142    2576 scope.go:117] "RemoveContainer" containerID="576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b"
Aug 04 18:02:38.345329 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:38.345315    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b\": container with ID starting with 576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b not found: ID does not exist" containerID="576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b"
Aug 04 18:02:38.345367 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.345332    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b"} err="failed to get container status \"576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b\": rpc error: code = NotFound desc = could not find container \"576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b\": container with ID starting with 576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b not found: ID does not exist"
Aug 04 18:02:38.345367 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.345345    2576 scope.go:117] "RemoveContainer" containerID="eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4"
Aug 04 18:02:38.345568 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:38.345551    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4\": container with ID starting with eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4 not found: ID does not exist" containerID="eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4"
Aug 04 18:02:38.345609 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.345572    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4"} err="failed to get container status \"eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4\": rpc error: code = NotFound desc = could not find container \"eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4\": container with ID starting with eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4 not found: ID does not exist"
Aug 04 18:02:38.345609 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.345585    2576 scope.go:117] "RemoveContainer" containerID="c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5"
Aug 04 18:02:38.345776 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:38.345763    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5\": container with ID starting with c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5 not found: ID does not exist" containerID="c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5"
Aug 04 18:02:38.345814 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.345778    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5"} err="failed to get container status \"c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5\": rpc error: code = NotFound desc = could not find container \"c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5\": container with ID starting with c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5 not found: ID does not exist"
Aug 04 18:02:38.345814 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.345789    2576 scope.go:117] "RemoveContainer" containerID="504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e"
Aug 04 18:02:38.345997 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:38.345980    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e\": container with ID starting with 504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e not found: ID does not exist" containerID="504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e"
Aug 04 18:02:38.346058 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.346005    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e"} err="failed to get container status \"504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e\": rpc error: code = NotFound desc = could not find container \"504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e\": container with ID starting with 504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e not found: ID does not exist"
Aug 04 18:02:38.346058 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.346025    2576 scope.go:117] "RemoveContainer" containerID="91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c"
Aug 04 18:02:38.346277 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.346258    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c"} err="failed to get container status \"91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c\": rpc error: code = NotFound desc = could not find container \"91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c\": container with ID starting with 91bfa9e650a0d112c5a78247f627ba578b097810c7592b6af51fad16f160426c not found: ID does not exist"
Aug 04 18:02:38.346328 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.346277    2576 scope.go:117] "RemoveContainer" containerID="cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce"
Aug 04 18:02:38.346478 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.346461    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce"} err="failed to get container status \"cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce\": rpc error: code = NotFound desc = could not find container \"cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce\": container with ID starting with cb792387eb60c22671f61d261efda0aed9825bf04fe1832c159c5cd7749905ce not found: ID does not exist"
Aug 04 18:02:38.346543 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.346481    2576 scope.go:117] "RemoveContainer" containerID="2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298"
Aug 04 18:02:38.346690 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.346673    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298"} err="failed to get container status \"2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298\": rpc error: code = NotFound desc = could not find container \"2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298\": container with ID starting with 2b85755e43bec72ba3d861b5d10420225cd05269fa1e53d5939bf9922ac6e298 not found: ID does not exist"
Aug 04 18:02:38.346746 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.346691    2576 scope.go:117] "RemoveContainer" containerID="576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b"
Aug 04 18:02:38.346875 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.346861    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b"} err="failed to get container status \"576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b\": rpc error: code = NotFound desc = could not find container \"576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b\": container with ID starting with 576f2814c31fba9829249044f585c555b82f1f4d2db8136c0e8290312264585b not found: ID does not exist"
Aug 04 18:02:38.346875 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.346874    2576 scope.go:117] "RemoveContainer" containerID="eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4"
Aug 04 18:02:38.347030 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.347015    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4"} err="failed to get container status \"eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4\": rpc error: code = NotFound desc = could not find container \"eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4\": container with ID starting with eeb5826c79ecf9b78c142e5270d26b887948e331cb5bb1341ee8eab81a319df4 not found: ID does not exist"
Aug 04 18:02:38.347072 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.347030    2576 scope.go:117] "RemoveContainer" containerID="c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5"
Aug 04 18:02:38.347227 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.347211    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5"} err="failed to get container status \"c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5\": rpc error: code = NotFound desc = could not find container \"c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5\": container with ID starting with c834aac63640acdaea1ac2b066064954d58f6a745d41826f61cc11b0cb5514b5 not found: ID does not exist"
Aug 04 18:02:38.347227 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.347226    2576 scope.go:117] "RemoveContainer" containerID="504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e"
Aug 04 18:02:38.347439 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.347420    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e"} err="failed to get container status \"504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e\": rpc error: code = NotFound desc = could not find container \"504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e\": container with ID starting with 504b48dcb8ff5e6b4c5cdb9460b671a370c2f4b00d7c2147e16d3980244a5a0e not found: ID does not exist"
Aug 04 18:02:38.352727 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.352702    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:02:38.353006 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.352995    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="716abfd2-ee03-4ebc-bb89-d41d82d26090" containerName="console"
Aug 04 18:02:38.353059 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353008    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="716abfd2-ee03-4ebc-bb89-d41d82d26090" containerName="console"
Aug 04 18:02:38.353059 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353019    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="init-config-reloader"
Aug 04 18:02:38.353059 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353025    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="init-config-reloader"
Aug 04 18:02:38.353059 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353033    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7da1159-037e-4155-8e4d-dab9a539e576" containerName="console"
Aug 04 18:02:38.353059 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353039    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7da1159-037e-4155-8e4d-dab9a539e576" containerName="console"
Aug 04 18:02:38.353059 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353045    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="kube-rbac-proxy"
Aug 04 18:02:38.353059 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353050    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="kube-rbac-proxy"
Aug 04 18:02:38.353059 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353057    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623" containerName="console"
Aug 04 18:02:38.353059 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353062    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623" containerName="console"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353070    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="config-reloader"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353076    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="config-reloader"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353083    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="alertmanager"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353087    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="alertmanager"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353093    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="kube-rbac-proxy-metric"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353098    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="kube-rbac-proxy-metric"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353126    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="kube-rbac-proxy-web"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353132    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="kube-rbac-proxy-web"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353137    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="prom-label-proxy"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353142    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="prom-label-proxy"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353187    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="alertmanager"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353194    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="config-reloader"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353201    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7da1159-037e-4155-8e4d-dab9a539e576" containerName="console"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353206    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="kube-rbac-proxy-web"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353212    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="kube-rbac-proxy-metric"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353217    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="716abfd2-ee03-4ebc-bb89-d41d82d26090" containerName="console"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353222    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="prom-label-proxy"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353228    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d4e3b5a-c43a-4b19-b7fe-3478a6b9b623" containerName="console"
Aug 04 18:02:38.353415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.353233    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" containerName="kube-rbac-proxy"
Aug 04 18:02:38.358824 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.358808    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.361620 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.361597    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Aug 04 18:02:38.361620 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.361608    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Aug 04 18:02:38.361924 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.361908    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Aug 04 18:02:38.362010 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.361929    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Aug 04 18:02:38.362010 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.361934    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Aug 04 18:02:38.362010 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.361957    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Aug 04 18:02:38.362273 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.362256    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d865b\""
Aug 04 18:02:38.362333 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.362273    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Aug 04 18:02:38.362333 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.362257    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Aug 04 18:02:38.370366 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.370221    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Aug 04 18:02:38.378424 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.378385    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:02:38.458070 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.458030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.458070 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.458075    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d366eb8e-383b-43f3-93a9-c133cedf8334-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.458324 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.458100    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d366eb8e-383b-43f3-93a9-c133cedf8334-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.458324 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.458172    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.458324 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.458213    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d366eb8e-383b-43f3-93a9-c133cedf8334-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.458324 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.458238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xfhw\" (UniqueName: \"kubernetes.io/projected/d366eb8e-383b-43f3-93a9-c133cedf8334-kube-api-access-4xfhw\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.458324 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.458258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-config-volume\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.458324 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.458278    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d366eb8e-383b-43f3-93a9-c133cedf8334-config-out\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.458530 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.458326    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.458530 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.458391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-web-config\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.458530 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.458422    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d366eb8e-383b-43f3-93a9-c133cedf8334-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.458530 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.458447    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.458530 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.458463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.559574 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.559482    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.559574 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.559530    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d366eb8e-383b-43f3-93a9-c133cedf8334-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.559574 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.559556    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d366eb8e-383b-43f3-93a9-c133cedf8334-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.559836 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.559592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.559836 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.559629    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d366eb8e-383b-43f3-93a9-c133cedf8334-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.559836 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.559665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xfhw\" (UniqueName: \"kubernetes.io/projected/d366eb8e-383b-43f3-93a9-c133cedf8334-kube-api-access-4xfhw\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.559836 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.559693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-config-volume\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.559836 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.559749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d366eb8e-383b-43f3-93a9-c133cedf8334-config-out\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.559836 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.559797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.559836 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.559832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-web-config\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.560267 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.559860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d366eb8e-383b-43f3-93a9-c133cedf8334-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.560267 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.559910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.560267 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.559936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.560267 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.560152    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d366eb8e-383b-43f3-93a9-c133cedf8334-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.560602 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.560573    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d366eb8e-383b-43f3-93a9-c133cedf8334-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.560774 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.560712    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d366eb8e-383b-43f3-93a9-c133cedf8334-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.562755 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.562612    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d366eb8e-383b-43f3-93a9-c133cedf8334-config-out\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.562755 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.562739    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.562932 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.562769    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.562932 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.562868    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-config-volume\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.563043 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.562977    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-web-config\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.563043 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.563003    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.563366 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.563348    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.563428 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.563409    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d366eb8e-383b-43f3-93a9-c133cedf8334-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.564809 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.564788    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d366eb8e-383b-43f3-93a9-c133cedf8334-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.569160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.569141    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xfhw\" (UniqueName: \"kubernetes.io/projected/d366eb8e-383b-43f3-93a9-c133cedf8334-kube-api-access-4xfhw\") pod \"alertmanager-main-0\" (UID: \"d366eb8e-383b-43f3-93a9-c133cedf8334\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.581380 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.581357    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aa34905-832c-4e14-a6b0-77304fc88782" path="/var/lib/kubelet/pods/9aa34905-832c-4e14-a6b0-77304fc88782/volumes"
Aug 04 18:02:38.673805 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.673776    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:38.816952 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:38.816923    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:02:38.818755 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:02:38.818728    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd366eb8e_383b_43f3_93a9_c133cedf8334.slice/crio-359752646c72332ea23e5fb278cf2386676ae0e7ebe94fdf157681ec20f2ea51 WatchSource:0}: Error finding container 359752646c72332ea23e5fb278cf2386676ae0e7ebe94fdf157681ec20f2ea51: Status 404 returned error can't find the container with id 359752646c72332ea23e5fb278cf2386676ae0e7ebe94fdf157681ec20f2ea51
Aug 04 18:02:39.296414 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:39.296381    2576 generic.go:356] "Generic (PLEG): container finished" podID="d366eb8e-383b-43f3-93a9-c133cedf8334" containerID="e18dac5aed699940c274ffc478bf66ed0910d9f16e6251a02c49459e2a0e8c69" exitCode=0
Aug 04 18:02:39.296581 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:39.296422    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d366eb8e-383b-43f3-93a9-c133cedf8334","Type":"ContainerDied","Data":"e18dac5aed699940c274ffc478bf66ed0910d9f16e6251a02c49459e2a0e8c69"}
Aug 04 18:02:39.296581 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:39.296472    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d366eb8e-383b-43f3-93a9-c133cedf8334","Type":"ContainerStarted","Data":"359752646c72332ea23e5fb278cf2386676ae0e7ebe94fdf157681ec20f2ea51"}
Aug 04 18:02:40.303715 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.303679    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d366eb8e-383b-43f3-93a9-c133cedf8334","Type":"ContainerStarted","Data":"e837b4a75082eec2aa2baea901729657d07b3985e1103eeac8be6a3445c219b4"}
Aug 04 18:02:40.304091 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.303746    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d366eb8e-383b-43f3-93a9-c133cedf8334","Type":"ContainerStarted","Data":"3ae8064e6dfe013e3341f789787670a99526d6bc2616f3529dd030e49a8c4005"}
Aug 04 18:02:40.304091 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.303776    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d366eb8e-383b-43f3-93a9-c133cedf8334","Type":"ContainerStarted","Data":"335ab4eb7d03441f24ea20c70c186c9eb7ee95427c48174d8582f2c38ef49820"}
Aug 04 18:02:40.304091 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.303801    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d366eb8e-383b-43f3-93a9-c133cedf8334","Type":"ContainerStarted","Data":"6f38861c6902c86c02d37db7aeeafba811cdd34587c456bc38764605dc98b2e0"}
Aug 04 18:02:40.304091 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.303823    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d366eb8e-383b-43f3-93a9-c133cedf8334","Type":"ContainerStarted","Data":"f114689dcfe81d45a20ff0e104c3558d4a5f0df9dbc68d5225f579ec7fff348b"}
Aug 04 18:02:40.304091 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.303845    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d366eb8e-383b-43f3-93a9-c133cedf8334","Type":"ContainerStarted","Data":"dfbddb0d7dcba5d4796a1e1827d3fbe3fa4991fa82d02c21a863ec05d8f67698"}
Aug 04 18:02:40.349953 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.349895    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.337636049 podStartE2EDuration="2.34987277s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.236721ms" isStatefulPod=false podCreationTimestamp="2026-08-04 18:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:40.349067146 +0000 UTC m=+160.324898844" watchObservedRunningTime="2026-08-04 18:02:40.34987277 +0000 UTC m=+160.325704467"
Aug 04 18:02:40.729941 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.729857    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:40.730322 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.730154    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="prometheus" containerID="cri-o://d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272" gracePeriod=600
Aug 04 18:02:40.730322 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.730194    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="kube-rbac-proxy" containerID="cri-o://eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6" gracePeriod=600
Aug 04 18:02:40.730322 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.730195    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="thanos-sidecar" containerID="cri-o://ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe" gracePeriod=600
Aug 04 18:02:40.730322 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.730218    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="config-reloader" containerID="cri-o://6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1" gracePeriod=600
Aug 04 18:02:40.730322 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.730270    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c" gracePeriod=600
Aug 04 18:02:40.730649 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.730213    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="kube-rbac-proxy-web" containerID="cri-o://387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943" gracePeriod=600
Aug 04 18:02:40.972452 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:40.972429    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.082781 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.082748    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-grpc-tls\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.082967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.082788    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-0\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.082967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.082811    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-tls\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.082967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.082831    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-1\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.082967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.082854    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-2\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.082967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.082894    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-kubelet-serving-ca-bundle\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.082967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.082921    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hx7z\" (UniqueName: \"kubernetes.io/projected/a3385a98-db22-462f-9f7b-4e7178fd3ef0-kube-api-access-4hx7z\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.082967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.082951    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-metrics-client-certs\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.083348 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.082991    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-config\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.083348 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083019    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.083348 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083049    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.083348 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083082    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-web-config\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.083348 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083161    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-trusted-ca-bundle\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.083348 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083200    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3385a98-db22-462f-9f7b-4e7178fd3ef0-tls-assets\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.083348 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083206    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:41.083348 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083226    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-db\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.083348 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083255    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-thanos-prometheus-http-client-file\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.083348 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083287    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-serving-certs-ca-bundle\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.083348 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083326    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3385a98-db22-462f-9f7b-4e7178fd3ef0-config-out\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.083896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083372    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-metrics-client-ca\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.083896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083381    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:41.083896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083375    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:41.083896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083422    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-kube-rbac-proxy\") pod \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\" (UID: \"a3385a98-db22-462f-9f7b-4e7178fd3ef0\") "
Aug 04 18:02:41.083896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083698    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.083896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083718    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.083896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.083733    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.084268 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.084071    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:41.084805 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.084484    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:41.084805 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.084758    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:41.085247 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.084894    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:02:41.085247 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.085182    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:41.086099 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.086059    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:41.086218 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.086127    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:41.087172 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.087132    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:41.087463 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.087438    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:41.087684 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.087665    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3385a98-db22-462f-9f7b-4e7178fd3ef0-kube-api-access-4hx7z" (OuterVolumeSpecName: "kube-api-access-4hx7z") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "kube-api-access-4hx7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:41.087766 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.087705    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3385a98-db22-462f-9f7b-4e7178fd3ef0-config-out" (OuterVolumeSpecName: "config-out") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:02:41.087819 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.087776    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:41.087860 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.087822    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-config" (OuterVolumeSpecName: "config") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:41.087894 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.087857    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:41.088016 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.088000    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3385a98-db22-462f-9f7b-4e7178fd3ef0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:41.088534 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.088517    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:41.098424 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.098400    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-web-config" (OuterVolumeSpecName: "web-config") pod "a3385a98-db22-462f-9f7b-4e7178fd3ef0" (UID: "a3385a98-db22-462f-9f7b-4e7178fd3ef0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:41.185061 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185022    2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185061 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185053    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185061 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185064    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185061 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185075    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-web-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185084    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185093    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3385a98-db22-462f-9f7b-4e7178fd3ef0-tls-assets\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185101    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-db\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185130    2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185139    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185147    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3385a98-db22-462f-9f7b-4e7178fd3ef0-config-out\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185155    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-configmap-metrics-client-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185163    2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-kube-rbac-proxy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185170    2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-grpc-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185178    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3385a98-db22-462f-9f7b-4e7178fd3ef0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185186    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185196    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hx7z\" (UniqueName: \"kubernetes.io/projected/a3385a98-db22-462f-9f7b-4e7178fd3ef0-kube-api-access-4hx7z\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.185355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.185204    2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3385a98-db22-462f-9f7b-4e7178fd3ef0-secret-metrics-client-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:02:41.311888 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.311849    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerID="427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c" exitCode=0
Aug 04 18:02:41.311888 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.311883    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerID="eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6" exitCode=0
Aug 04 18:02:41.311888 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.311892    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerID="387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943" exitCode=0
Aug 04 18:02:41.312419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.311901    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerID="ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe" exitCode=0
Aug 04 18:02:41.312419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.311909    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerID="6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1" exitCode=0
Aug 04 18:02:41.312419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.311916    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerID="d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272" exitCode=0
Aug 04 18:02:41.312419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.311947    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.312419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.311936    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerDied","Data":"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"}
Aug 04 18:02:41.312419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.312156    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerDied","Data":"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"}
Aug 04 18:02:41.312419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.312191    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerDied","Data":"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"}
Aug 04 18:02:41.312419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.312217    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerDied","Data":"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"}
Aug 04 18:02:41.312419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.312243    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerDied","Data":"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"}
Aug 04 18:02:41.312419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.312244    2576 scope.go:117] "RemoveContainer" containerID="427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"
Aug 04 18:02:41.312419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.312270    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerDied","Data":"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"}
Aug 04 18:02:41.312419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.312294    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3385a98-db22-462f-9f7b-4e7178fd3ef0","Type":"ContainerDied","Data":"3302412aceb7f83fc26e44a87d8c0d54bb23e49a9859ffc230aebaca43a4c95b"}
Aug 04 18:02:41.320452 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.320437    2576 scope.go:117] "RemoveContainer" containerID="eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"
Aug 04 18:02:41.327293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.327274    2576 scope.go:117] "RemoveContainer" containerID="387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"
Aug 04 18:02:41.334174 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.334156    2576 scope.go:117] "RemoveContainer" containerID="ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"
Aug 04 18:02:41.338935 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.338909    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:41.341408 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.341391    2576 scope.go:117] "RemoveContainer" containerID="6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"
Aug 04 18:02:41.343787 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.343765    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:41.348556 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.348538    2576 scope.go:117] "RemoveContainer" containerID="d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"
Aug 04 18:02:41.355479 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.355464    2576 scope.go:117] "RemoveContainer" containerID="789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"
Aug 04 18:02:41.362475 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.362453    2576 scope.go:117] "RemoveContainer" containerID="427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"
Aug 04 18:02:41.362763 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:41.362741    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c\": container with ID starting with 427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c not found: ID does not exist" containerID="427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"
Aug 04 18:02:41.362820 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.362771    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"} err="failed to get container status \"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c\": rpc error: code = NotFound desc = could not find container \"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c\": container with ID starting with 427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c not found: ID does not exist"
Aug 04 18:02:41.362820 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.362790    2576 scope.go:117] "RemoveContainer" containerID="eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"
Aug 04 18:02:41.363026 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:41.363011    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6\": container with ID starting with eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6 not found: ID does not exist" containerID="eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"
Aug 04 18:02:41.363063 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.363030    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"} err="failed to get container status \"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6\": rpc error: code = NotFound desc = could not find container \"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6\": container with ID starting with eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6 not found: ID does not exist"
Aug 04 18:02:41.363063 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.363045    2576 scope.go:117] "RemoveContainer" containerID="387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"
Aug 04 18:02:41.363271 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:41.363254    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943\": container with ID starting with 387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943 not found: ID does not exist" containerID="387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"
Aug 04 18:02:41.363335 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.363273    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"} err="failed to get container status \"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943\": rpc error: code = NotFound desc = could not find container \"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943\": container with ID starting with 387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943 not found: ID does not exist"
Aug 04 18:02:41.363335 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.363284    2576 scope.go:117] "RemoveContainer" containerID="ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"
Aug 04 18:02:41.363477 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:41.363464    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe\": container with ID starting with ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe not found: ID does not exist" containerID="ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"
Aug 04 18:02:41.363516 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.363480    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"} err="failed to get container status \"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe\": rpc error: code = NotFound desc = could not find container \"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe\": container with ID starting with ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe not found: ID does not exist"
Aug 04 18:02:41.363516 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.363491    2576 scope.go:117] "RemoveContainer" containerID="6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"
Aug 04 18:02:41.363702 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:41.363684    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1\": container with ID starting with 6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1 not found: ID does not exist" containerID="6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"
Aug 04 18:02:41.363742 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.363707    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"} err="failed to get container status \"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1\": rpc error: code = NotFound desc = could not find container \"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1\": container with ID starting with 6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1 not found: ID does not exist"
Aug 04 18:02:41.363742 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.363725    2576 scope.go:117] "RemoveContainer" containerID="d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"
Aug 04 18:02:41.363957 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:41.363940    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272\": container with ID starting with d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272 not found: ID does not exist" containerID="d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"
Aug 04 18:02:41.363994 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.363962    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"} err="failed to get container status \"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272\": rpc error: code = NotFound desc = could not find container \"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272\": container with ID starting with d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272 not found: ID does not exist"
Aug 04 18:02:41.363994 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.363977    2576 scope.go:117] "RemoveContainer" containerID="789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"
Aug 04 18:02:41.364460 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:02:41.364442    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5\": container with ID starting with 789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5 not found: ID does not exist" containerID="789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"
Aug 04 18:02:41.364511 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.364462    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"} err="failed to get container status \"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5\": rpc error: code = NotFound desc = could not find container \"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5\": container with ID starting with 789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5 not found: ID does not exist"
Aug 04 18:02:41.364511 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.364474    2576 scope.go:117] "RemoveContainer" containerID="427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"
Aug 04 18:02:41.364673 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.364656    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"} err="failed to get container status \"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c\": rpc error: code = NotFound desc = could not find container \"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c\": container with ID starting with 427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c not found: ID does not exist"
Aug 04 18:02:41.364718 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.364674    2576 scope.go:117] "RemoveContainer" containerID="eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"
Aug 04 18:02:41.364850 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.364833    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"} err="failed to get container status \"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6\": rpc error: code = NotFound desc = could not find container \"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6\": container with ID starting with eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6 not found: ID does not exist"
Aug 04 18:02:41.364892 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.364850    2576 scope.go:117] "RemoveContainer" containerID="387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"
Aug 04 18:02:41.365067 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.365052    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"} err="failed to get container status \"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943\": rpc error: code = NotFound desc = could not find container \"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943\": container with ID starting with 387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943 not found: ID does not exist"
Aug 04 18:02:41.365103 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.365067    2576 scope.go:117] "RemoveContainer" containerID="ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"
Aug 04 18:02:41.365314 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.365294    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"} err="failed to get container status \"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe\": rpc error: code = NotFound desc = could not find container \"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe\": container with ID starting with ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe not found: ID does not exist"
Aug 04 18:02:41.365392 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.365315    2576 scope.go:117] "RemoveContainer" containerID="6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"
Aug 04 18:02:41.365577 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.365533    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"} err="failed to get container status \"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1\": rpc error: code = NotFound desc = could not find container \"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1\": container with ID starting with 6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1 not found: ID does not exist"
Aug 04 18:02:41.365577 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.365566    2576 scope.go:117] "RemoveContainer" containerID="d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"
Aug 04 18:02:41.365967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.365843    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"} err="failed to get container status \"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272\": rpc error: code = NotFound desc = could not find container \"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272\": container with ID starting with d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272 not found: ID does not exist"
Aug 04 18:02:41.365967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.365868    2576 scope.go:117] "RemoveContainer" containerID="789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"
Aug 04 18:02:41.366143 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.366122    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"} err="failed to get container status \"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5\": rpc error: code = NotFound desc = could not find container \"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5\": container with ID starting with 789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5 not found: ID does not exist"
Aug 04 18:02:41.366198 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.366146    2576 scope.go:117] "RemoveContainer" containerID="427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"
Aug 04 18:02:41.366388 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.366371    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"} err="failed to get container status \"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c\": rpc error: code = NotFound desc = could not find container \"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c\": container with ID starting with 427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c not found: ID does not exist"
Aug 04 18:02:41.366388 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.366387    2576 scope.go:117] "RemoveContainer" containerID="eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"
Aug 04 18:02:41.366600 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.366581    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"} err="failed to get container status \"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6\": rpc error: code = NotFound desc = could not find container \"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6\": container with ID starting with eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6 not found: ID does not exist"
Aug 04 18:02:41.366638 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.366601    2576 scope.go:117] "RemoveContainer" containerID="387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"
Aug 04 18:02:41.366801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.366786    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"} err="failed to get container status \"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943\": rpc error: code = NotFound desc = could not find container \"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943\": container with ID starting with 387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943 not found: ID does not exist"
Aug 04 18:02:41.366801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.366800    2576 scope.go:117] "RemoveContainer" containerID="ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"
Aug 04 18:02:41.367009 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.366991    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"} err="failed to get container status \"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe\": rpc error: code = NotFound desc = could not find container \"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe\": container with ID starting with ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe not found: ID does not exist"
Aug 04 18:02:41.367054 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.367010    2576 scope.go:117] "RemoveContainer" containerID="6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"
Aug 04 18:02:41.367242 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.367224    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"} err="failed to get container status \"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1\": rpc error: code = NotFound desc = could not find container \"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1\": container with ID starting with 6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1 not found: ID does not exist"
Aug 04 18:02:41.367242 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.367239    2576 scope.go:117] "RemoveContainer" containerID="d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"
Aug 04 18:02:41.367456 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.367438    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"} err="failed to get container status \"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272\": rpc error: code = NotFound desc = could not find container \"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272\": container with ID starting with d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272 not found: ID does not exist"
Aug 04 18:02:41.367498 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.367458    2576 scope.go:117] "RemoveContainer" containerID="789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"
Aug 04 18:02:41.367686 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.367667    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"} err="failed to get container status \"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5\": rpc error: code = NotFound desc = could not find container \"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5\": container with ID starting with 789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5 not found: ID does not exist"
Aug 04 18:02:41.367729 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.367686    2576 scope.go:117] "RemoveContainer" containerID="427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"
Aug 04 18:02:41.367913 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.367899    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"} err="failed to get container status \"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c\": rpc error: code = NotFound desc = could not find container \"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c\": container with ID starting with 427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c not found: ID does not exist"
Aug 04 18:02:41.367954 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.367913    2576 scope.go:117] "RemoveContainer" containerID="eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"
Aug 04 18:02:41.368146 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.368105    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"} err="failed to get container status \"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6\": rpc error: code = NotFound desc = could not find container \"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6\": container with ID starting with eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6 not found: ID does not exist"
Aug 04 18:02:41.368146 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.368144    2576 scope.go:117] "RemoveContainer" containerID="387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"
Aug 04 18:02:41.368378 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.368362    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"} err="failed to get container status \"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943\": rpc error: code = NotFound desc = could not find container \"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943\": container with ID starting with 387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943 not found: ID does not exist"
Aug 04 18:02:41.368420 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.368379    2576 scope.go:117] "RemoveContainer" containerID="ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"
Aug 04 18:02:41.368614 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.368595    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"} err="failed to get container status \"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe\": rpc error: code = NotFound desc = could not find container \"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe\": container with ID starting with ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe not found: ID does not exist"
Aug 04 18:02:41.368682 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.368617    2576 scope.go:117] "RemoveContainer" containerID="6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"
Aug 04 18:02:41.368840 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.368825    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"} err="failed to get container status \"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1\": rpc error: code = NotFound desc = could not find container \"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1\": container with ID starting with 6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1 not found: ID does not exist"
Aug 04 18:02:41.368881 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.368841    2576 scope.go:117] "RemoveContainer" containerID="d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"
Aug 04 18:02:41.369023 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.369005    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"} err="failed to get container status \"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272\": rpc error: code = NotFound desc = could not find container \"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272\": container with ID starting with d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272 not found: ID does not exist"
Aug 04 18:02:41.369071 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.369024    2576 scope.go:117] "RemoveContainer" containerID="789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"
Aug 04 18:02:41.369258 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.369242    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"} err="failed to get container status \"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5\": rpc error: code = NotFound desc = could not find container \"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5\": container with ID starting with 789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5 not found: ID does not exist"
Aug 04 18:02:41.369307 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.369257    2576 scope.go:117] "RemoveContainer" containerID="427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"
Aug 04 18:02:41.369442 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.369426    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"} err="failed to get container status \"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c\": rpc error: code = NotFound desc = could not find container \"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c\": container with ID starting with 427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c not found: ID does not exist"
Aug 04 18:02:41.369477 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.369445    2576 scope.go:117] "RemoveContainer" containerID="eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"
Aug 04 18:02:41.369651 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.369636    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"} err="failed to get container status \"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6\": rpc error: code = NotFound desc = could not find container \"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6\": container with ID starting with eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6 not found: ID does not exist"
Aug 04 18:02:41.369651 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.369650    2576 scope.go:117] "RemoveContainer" containerID="387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"
Aug 04 18:02:41.369849 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.369823    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"} err="failed to get container status \"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943\": rpc error: code = NotFound desc = could not find container \"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943\": container with ID starting with 387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943 not found: ID does not exist"
Aug 04 18:02:41.369888 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.369850    2576 scope.go:117] "RemoveContainer" containerID="ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"
Aug 04 18:02:41.370049 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.370028    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"} err="failed to get container status \"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe\": rpc error: code = NotFound desc = could not find container \"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe\": container with ID starting with ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe not found: ID does not exist"
Aug 04 18:02:41.370127 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.370060    2576 scope.go:117] "RemoveContainer" containerID="6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"
Aug 04 18:02:41.370298 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.370280    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"} err="failed to get container status \"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1\": rpc error: code = NotFound desc = could not find container \"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1\": container with ID starting with 6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1 not found: ID does not exist"
Aug 04 18:02:41.370298 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.370297    2576 scope.go:117] "RemoveContainer" containerID="d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"
Aug 04 18:02:41.370513 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.370495    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"} err="failed to get container status \"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272\": rpc error: code = NotFound desc = could not find container \"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272\": container with ID starting with d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272 not found: ID does not exist"
Aug 04 18:02:41.370561 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.370514    2576 scope.go:117] "RemoveContainer" containerID="789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"
Aug 04 18:02:41.370697 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.370681    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"} err="failed to get container status \"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5\": rpc error: code = NotFound desc = could not find container \"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5\": container with ID starting with 789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5 not found: ID does not exist"
Aug 04 18:02:41.370744 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.370697    2576 scope.go:117] "RemoveContainer" containerID="427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"
Aug 04 18:02:41.370885 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.370866    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c"} err="failed to get container status \"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c\": rpc error: code = NotFound desc = could not find container \"427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c\": container with ID starting with 427080f93cd965148773060b3d48d3d9337bc7b45b9236ecfcacaf12e44d209c not found: ID does not exist"
Aug 04 18:02:41.370931 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.370887    2576 scope.go:117] "RemoveContainer" containerID="eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"
Aug 04 18:02:41.371070 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.371054    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6"} err="failed to get container status \"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6\": rpc error: code = NotFound desc = could not find container \"eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6\": container with ID starting with eef67191f06b379754c6226d5a38745b0e7e7debd6049bbbb76249c9acc916c6 not found: ID does not exist"
Aug 04 18:02:41.371136 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.371071    2576 scope.go:117] "RemoveContainer" containerID="387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"
Aug 04 18:02:41.371285 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.371269    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943"} err="failed to get container status \"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943\": rpc error: code = NotFound desc = could not find container \"387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943\": container with ID starting with 387139e97e10cd73a92c90829ebe797008d84f130e9698f08b8110ea1c29f943 not found: ID does not exist"
Aug 04 18:02:41.371355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.371283    2576 scope.go:117] "RemoveContainer" containerID="ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"
Aug 04 18:02:41.371521 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.371503    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe"} err="failed to get container status \"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe\": rpc error: code = NotFound desc = could not find container \"ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe\": container with ID starting with ba39b2bca8c2a7491a7823414a1f6aa60447daad07c61ca3ca6db5d4b7035fbe not found: ID does not exist"
Aug 04 18:02:41.371585 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.371522    2576 scope.go:117] "RemoveContainer" containerID="6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"
Aug 04 18:02:41.371738 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.371722    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1"} err="failed to get container status \"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1\": rpc error: code = NotFound desc = could not find container \"6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1\": container with ID starting with 6ae338a8329094b17e0d9c1cbae18e0ffca2903e2045218c7e76697d981075c1 not found: ID does not exist"
Aug 04 18:02:41.371789 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.371738    2576 scope.go:117] "RemoveContainer" containerID="d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"
Aug 04 18:02:41.371939 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.371924    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272"} err="failed to get container status \"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272\": rpc error: code = NotFound desc = could not find container \"d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272\": container with ID starting with d4f72ef5fff9912a9858631ad03932ed63466e6677a9d23817b68edcbbf34272 not found: ID does not exist"
Aug 04 18:02:41.372000 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.371939    2576 scope.go:117] "RemoveContainer" containerID="789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"
Aug 04 18:02:41.372181 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.372165    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5"} err="failed to get container status \"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5\": rpc error: code = NotFound desc = could not find container \"789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5\": container with ID starting with 789dc5b7de9ab36e8b428dd9649d32b356ee948d58f4a5da827d8f12ce2caac5 not found: ID does not exist"
Aug 04 18:02:41.380023 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380001    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:41.380315 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380303    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="kube-rbac-proxy"
Aug 04 18:02:41.380355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380317    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="kube-rbac-proxy"
Aug 04 18:02:41.380355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380328    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="init-config-reloader"
Aug 04 18:02:41.380355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380333    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="init-config-reloader"
Aug 04 18:02:41.380355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380343    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="kube-rbac-proxy-thanos"
Aug 04 18:02:41.380355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380348    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="kube-rbac-proxy-thanos"
Aug 04 18:02:41.380355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380355    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="prometheus"
Aug 04 18:02:41.380523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380361    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="prometheus"
Aug 04 18:02:41.380523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380367    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="thanos-sidecar"
Aug 04 18:02:41.380523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380372    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="thanos-sidecar"
Aug 04 18:02:41.380523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380379    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="kube-rbac-proxy-web"
Aug 04 18:02:41.380523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380384    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="kube-rbac-proxy-web"
Aug 04 18:02:41.380523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380393    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="config-reloader"
Aug 04 18:02:41.380523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380398    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="config-reloader"
Aug 04 18:02:41.380523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380444    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="config-reloader"
Aug 04 18:02:41.380523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380453    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="kube-rbac-proxy-web"
Aug 04 18:02:41.380523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380459    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="kube-rbac-proxy-thanos"
Aug 04 18:02:41.380523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380466    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="thanos-sidecar"
Aug 04 18:02:41.380523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380471    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="prometheus"
Aug 04 18:02:41.380523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.380477    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" containerName="kube-rbac-proxy"
Aug 04 18:02:41.384790 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.384772    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.387246 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.387229    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-brzkw\""
Aug 04 18:02:41.387575 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.387561    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Aug 04 18:02:41.387641 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.387561    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Aug 04 18:02:41.387822 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.387805    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Aug 04 18:02:41.387922 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.387907    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Aug 04 18:02:41.387978 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.387940    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-djou5n680khp2\""
Aug 04 18:02:41.388035 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.387996    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Aug 04 18:02:41.388084 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.388036    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Aug 04 18:02:41.388084 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.388048    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Aug 04 18:02:41.388262 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.388125    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Aug 04 18:02:41.388389 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.388372    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Aug 04 18:02:41.388452 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.388390    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Aug 04 18:02:41.388645 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.388631    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Aug 04 18:02:41.388903 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.388889    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Aug 04 18:02:41.390786 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.390766    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Aug 04 18:02:41.398508 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.398475    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Aug 04 18:02:41.408249 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.408217    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:41.487244 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487244 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-config\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/163345cf-fd0e-4d1b-a2c4-aef129b9f525-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487361    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487452    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487700 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/163345cf-fd0e-4d1b-a2c4-aef129b9f525-config-out\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487700 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487700 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487551    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487700 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487592    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487700 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487700 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487625    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8qkz\" (UniqueName: \"kubernetes.io/projected/163345cf-fd0e-4d1b-a2c4-aef129b9f525-kube-api-access-q8qkz\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487700 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-web-config\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487700 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487700 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487985 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.487985 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.487735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.588455 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.588354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.588455 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.588401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.588455 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.588425    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8qkz\" (UniqueName: \"kubernetes.io/projected/163345cf-fd0e-4d1b-a2c4-aef129b9f525-kube-api-access-q8qkz\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.588455 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.588457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-web-config\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.588777 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.588480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.588777 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.588503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.588777 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.588530    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.588941 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.588811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.588941 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.588867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.588941 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.588908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-config\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.588941 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.588939    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.589162 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.588973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/163345cf-fd0e-4d1b-a2c4-aef129b9f525-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.589162 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.588994    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.589162 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.589009    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.589162 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.589019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.589162 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.589058    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.589162 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.589104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.589162 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.589155    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.589484 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.589191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/163345cf-fd0e-4d1b-a2c4-aef129b9f525-config-out\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.589484 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.589231    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.589484 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.589246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.589484 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.589300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.589685 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.589562    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.589875 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.589849    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.589875 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.589865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.590604 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.590574    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.591611 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.591581    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.591711 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.591658    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-web-config\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.591835 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.591810    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.591901 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.591815    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.592281 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.592255    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.592577 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.592553    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/163345cf-fd0e-4d1b-a2c4-aef129b9f525-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.592661 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.592629    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.593167 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.593144    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.593254 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.593189    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.593718 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.593695    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/163345cf-fd0e-4d1b-a2c4-aef129b9f525-config-out\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.594101 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.594081    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.594428 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.594411    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/163345cf-fd0e-4d1b-a2c4-aef129b9f525-config\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.595015 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.594999    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/163345cf-fd0e-4d1b-a2c4-aef129b9f525-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.601299 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.601276    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8qkz\" (UniqueName: \"kubernetes.io/projected/163345cf-fd0e-4d1b-a2c4-aef129b9f525-kube-api-access-q8qkz\") pod \"prometheus-k8s-0\" (UID: \"163345cf-fd0e-4d1b-a2c4-aef129b9f525\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.699479 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.699446    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:41.845246 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:41.844891    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:41.847469 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:02:41.847441    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163345cf_fd0e_4d1b_a2c4_aef129b9f525.slice/crio-4e220eb31891aa09ddc9bae713b4615439d3983cafc9e6086a67228645460f68 WatchSource:0}: Error finding container 4e220eb31891aa09ddc9bae713b4615439d3983cafc9e6086a67228645460f68: Status 404 returned error can't find the container with id 4e220eb31891aa09ddc9bae713b4615439d3983cafc9e6086a67228645460f68
Aug 04 18:02:42.318298 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:42.318262    2576 generic.go:356] "Generic (PLEG): container finished" podID="163345cf-fd0e-4d1b-a2c4-aef129b9f525" containerID="01cbd99ce2ad6a3e81c5cae63680ece12888e81cc3c83ef95be2d710293255c6" exitCode=0
Aug 04 18:02:42.318695 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:42.318346    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163345cf-fd0e-4d1b-a2c4-aef129b9f525","Type":"ContainerDied","Data":"01cbd99ce2ad6a3e81c5cae63680ece12888e81cc3c83ef95be2d710293255c6"}
Aug 04 18:02:42.318695 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:42.318407    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163345cf-fd0e-4d1b-a2c4-aef129b9f525","Type":"ContainerStarted","Data":"4e220eb31891aa09ddc9bae713b4615439d3983cafc9e6086a67228645460f68"}
Aug 04 18:02:42.583407 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:42.583383    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3385a98-db22-462f-9f7b-4e7178fd3ef0" path="/var/lib/kubelet/pods/a3385a98-db22-462f-9f7b-4e7178fd3ef0/volumes"
Aug 04 18:02:43.329515 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:43.329481    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163345cf-fd0e-4d1b-a2c4-aef129b9f525","Type":"ContainerStarted","Data":"664e68f8942bf8e8f59813e52118ab7be99b5ef1fc52a6eb2b936cd1ee0d5e07"}
Aug 04 18:02:43.329928 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:43.329545    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163345cf-fd0e-4d1b-a2c4-aef129b9f525","Type":"ContainerStarted","Data":"87f25e9b2fe7ac869734cfd0c28d054562c58accdd0db106bed0d72ba18a551c"}
Aug 04 18:02:43.329928 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:43.329572    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163345cf-fd0e-4d1b-a2c4-aef129b9f525","Type":"ContainerStarted","Data":"8c57fd3e68551a7b9a77f5b9be1daa554b8250aa56874502540e533400593b6e"}
Aug 04 18:02:43.329928 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:43.329597    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163345cf-fd0e-4d1b-a2c4-aef129b9f525","Type":"ContainerStarted","Data":"29129e4f55650ada38558de1532379d3c9b264411469a666bc06a3a406276220"}
Aug 04 18:02:43.329928 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:43.329621    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163345cf-fd0e-4d1b-a2c4-aef129b9f525","Type":"ContainerStarted","Data":"3d54e750def81e9013bdcf75c24f6ca5c4f4ee7875d11da40a5f43b6e9818ec7"}
Aug 04 18:02:43.329928 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:43.329643    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163345cf-fd0e-4d1b-a2c4-aef129b9f525","Type":"ContainerStarted","Data":"59245fb389ca0f94adbe2557060bec2dce50090b4510be4869f187befae31680"}
Aug 04 18:02:43.366769 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:43.366709    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.346976186 podStartE2EDuration="2.366689327s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.713141ms" isStatefulPod=false podCreationTimestamp="2026-08-04 18:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:43.36419642 +0000 UTC m=+163.340028117" watchObservedRunningTime="2026-08-04 18:02:43.366689327 +0000 UTC m=+163.342521025"
Aug 04 18:02:44.122073 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:44.122033    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:44.218148 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:44.218088    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:45.143743 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:45.143708    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:45.244036 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:45.243998    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:45.344177 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:45.344142    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:46.700059 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:46.700018    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:47.926612 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:47.926576    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:48.064257 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:02:48.064222    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:41.699854 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:03:41.699811    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:03:41.715448 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:03:41.715418    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:03:42.583051 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:03:42.583021    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:03:59.585937 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:03:59.585902    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:59.685848 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:03:59.685806    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:59.747539 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:03:59.747510    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:04:39.997282 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:04:39.997201    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:04:40.089304 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:04:40.089267    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:04:46.077885 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:04:46.077841    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:04:46.316052 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:04:46.316020    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:04:46.951155 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:04:46.951104    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:04:47.553234 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:04:47.553202    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:04:47.649608 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:04:47.649567    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:00.431780 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:00.431752    2576 kubelet.go:1648] "Image garbage collection succeeded"
Aug 04 18:05:38.516535 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:38.516502    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:39.954127 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:39.954087    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:40.466314 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:40.466285    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:40.540452 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:40.540420    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:41.269312 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:41.269284    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:41.370734 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:41.370703    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:46.328662 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:46.328628    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:46.431466 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:46.431433    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:52.477586 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:52.477485    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:53.356386 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:53.356345    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:56.085278 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:56.085233    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:56.186162 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:56.186102    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:58.344270 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:58.344237    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:58.440747 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:05:58.440710    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:00.716864 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:00.716834    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:01.446907 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.446868    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cr8p2"]
Aug 04 18:06:01.450168 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.450146    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cr8p2"
Aug 04 18:06:01.452934 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.452912    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Aug 04 18:06:01.453840 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.453821    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Aug 04 18:06:01.453944 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.453821    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mdf8c\""
Aug 04 18:06:01.462743 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.462717    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cr8p2"]
Aug 04 18:06:01.484290 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.484257    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxb2\" (UniqueName: \"kubernetes.io/projected/d4589203-bc69-4df9-9eab-ae5a2d12b873-kube-api-access-7jxb2\") pod \"cert-manager-cainjector-559cdd8c87-cr8p2\" (UID: \"d4589203-bc69-4df9-9eab-ae5a2d12b873\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cr8p2"
Aug 04 18:06:01.484420 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.484386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4589203-bc69-4df9-9eab-ae5a2d12b873-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cr8p2\" (UID: \"d4589203-bc69-4df9-9eab-ae5a2d12b873\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cr8p2"
Aug 04 18:06:01.585419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.585384    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4589203-bc69-4df9-9eab-ae5a2d12b873-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cr8p2\" (UID: \"d4589203-bc69-4df9-9eab-ae5a2d12b873\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cr8p2"
Aug 04 18:06:01.585419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.585426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxb2\" (UniqueName: \"kubernetes.io/projected/d4589203-bc69-4df9-9eab-ae5a2d12b873-kube-api-access-7jxb2\") pod \"cert-manager-cainjector-559cdd8c87-cr8p2\" (UID: \"d4589203-bc69-4df9-9eab-ae5a2d12b873\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cr8p2"
Aug 04 18:06:01.593672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.593640    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4589203-bc69-4df9-9eab-ae5a2d12b873-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cr8p2\" (UID: \"d4589203-bc69-4df9-9eab-ae5a2d12b873\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cr8p2"
Aug 04 18:06:01.593842 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.593820    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxb2\" (UniqueName: \"kubernetes.io/projected/d4589203-bc69-4df9-9eab-ae5a2d12b873-kube-api-access-7jxb2\") pod \"cert-manager-cainjector-559cdd8c87-cr8p2\" (UID: \"d4589203-bc69-4df9-9eab-ae5a2d12b873\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cr8p2"
Aug 04 18:06:01.740949 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.740866    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:01.766889 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.766857    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cr8p2"
Aug 04 18:06:01.835484 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.835448    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:01.891984 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.891960    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cr8p2"]
Aug 04 18:06:01.894658 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:06:01.894630    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4589203_bc69_4df9_9eab_ae5a2d12b873.slice/crio-35f5c9aa2dddac2e56698e687077f580510aa2325dc2b17934fb61f69b54782f WatchSource:0}: Error finding container 35f5c9aa2dddac2e56698e687077f580510aa2325dc2b17934fb61f69b54782f: Status 404 returned error can't find the container with id 35f5c9aa2dddac2e56698e687077f580510aa2325dc2b17934fb61f69b54782f
Aug 04 18:06:01.896617 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:01.896601    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:06:02.104988 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:02.104961    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cr8p2" event={"ID":"d4589203-bc69-4df9-9eab-ae5a2d12b873","Type":"ContainerStarted","Data":"35f5c9aa2dddac2e56698e687077f580510aa2325dc2b17934fb61f69b54782f"}
Aug 04 18:06:02.838082 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:02.838043    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:02.932452 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:02.932424    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:04.243889 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:04.243854    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8ccxc"]
Aug 04 18:06:04.247677 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:04.247648    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8ccxc"
Aug 04 18:06:04.250297 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:04.250268    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-74tkt\""
Aug 04 18:06:04.259607 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:04.259330    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8ccxc"]
Aug 04 18:06:04.308442 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:04.308401    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bae09bcd-8cb1-44eb-9829-02083ee2f9eb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8ccxc\" (UID: \"bae09bcd-8cb1-44eb-9829-02083ee2f9eb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8ccxc"
Aug 04 18:06:04.308661 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:04.308453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6s4m\" (UniqueName: \"kubernetes.io/projected/bae09bcd-8cb1-44eb-9829-02083ee2f9eb-kube-api-access-q6s4m\") pod \"cert-manager-webhook-6486ffd997-8ccxc\" (UID: \"bae09bcd-8cb1-44eb-9829-02083ee2f9eb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8ccxc"
Aug 04 18:06:04.409891 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:04.409848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bae09bcd-8cb1-44eb-9829-02083ee2f9eb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8ccxc\" (UID: \"bae09bcd-8cb1-44eb-9829-02083ee2f9eb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8ccxc"
Aug 04 18:06:04.409891 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:04.409902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6s4m\" (UniqueName: \"kubernetes.io/projected/bae09bcd-8cb1-44eb-9829-02083ee2f9eb-kube-api-access-q6s4m\") pod \"cert-manager-webhook-6486ffd997-8ccxc\" (UID: \"bae09bcd-8cb1-44eb-9829-02083ee2f9eb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8ccxc"
Aug 04 18:06:04.420845 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:04.420809    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bae09bcd-8cb1-44eb-9829-02083ee2f9eb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8ccxc\" (UID: \"bae09bcd-8cb1-44eb-9829-02083ee2f9eb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8ccxc"
Aug 04 18:06:04.421056 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:04.421014    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6s4m\" (UniqueName: \"kubernetes.io/projected/bae09bcd-8cb1-44eb-9829-02083ee2f9eb-kube-api-access-q6s4m\") pod \"cert-manager-webhook-6486ffd997-8ccxc\" (UID: \"bae09bcd-8cb1-44eb-9829-02083ee2f9eb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8ccxc"
Aug 04 18:06:04.562704 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:04.562643    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8ccxc"
Aug 04 18:06:04.729970 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:04.729945    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8ccxc"]
Aug 04 18:06:06.279096 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:06:06.279045    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbae09bcd_8cb1_44eb_9829_02083ee2f9eb.slice/crio-9b730ad4fa26bec95368373d292856fec874b785e681f523947ee885847f7022 WatchSource:0}: Error finding container 9b730ad4fa26bec95368373d292856fec874b785e681f523947ee885847f7022: Status 404 returned error can't find the container with id 9b730ad4fa26bec95368373d292856fec874b785e681f523947ee885847f7022
Aug 04 18:06:07.129626 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:07.129584    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8ccxc" event={"ID":"bae09bcd-8cb1-44eb-9829-02083ee2f9eb","Type":"ContainerStarted","Data":"0117a812be865d75af62225a351f51f60b027fe5c77ec5f5bea47403330c1cbd"}
Aug 04 18:06:07.129807 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:07.129745    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8ccxc"
Aug 04 18:06:07.129807 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:07.129787    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8ccxc" event={"ID":"bae09bcd-8cb1-44eb-9829-02083ee2f9eb","Type":"ContainerStarted","Data":"9b730ad4fa26bec95368373d292856fec874b785e681f523947ee885847f7022"}
Aug 04 18:06:07.131145 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:07.131100    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cr8p2" event={"ID":"d4589203-bc69-4df9-9eab-ae5a2d12b873","Type":"ContainerStarted","Data":"152f3a735b4650017fbfae5b855fee01d907b29d345e01301f6d081f8a1ffe26"}
Aug 04 18:06:07.149586 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:07.149509    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-8ccxc" podStartSLOduration=2.539749291 podStartE2EDuration="3.149496823s" totalImagesPullingTime="609.747532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:06:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:06:07.148889886 +0000 UTC m=+367.124721585" watchObservedRunningTime="2026-08-04 18:06:07.149496823 +0000 UTC m=+367.125328520"
Aug 04 18:06:07.166672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:07.166632    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cr8p2" podStartSLOduration=1.7217969929999999 podStartE2EDuration="6.166619435s" totalImagesPullingTime="4.444822442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:06:07.165999283 +0000 UTC m=+367.141830981" watchObservedRunningTime="2026-08-04 18:06:07.166619435 +0000 UTC m=+367.142451132"
Aug 04 18:06:07.230833 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:07.230797    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:07.298989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:07.298945    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:08.319998 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:08.319957    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:13.137413 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:13.137384    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8ccxc"
Aug 04 18:06:14.298103 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:14.298065    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:14.395073 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:14.395037    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:15.945394 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:15.945357    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8dkb"]
Aug 04 18:06:15.948850 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:15.948823    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8dkb"
Aug 04 18:06:15.951833 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:15.951815    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-k99md\""
Aug 04 18:06:15.952377 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:15.952360    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:06:15.954025 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:15.954004    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Aug 04 18:06:16.013466 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:16.013425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/742530c4-d506-4393-a9e7-709de7b8580e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q8dkb\" (UID: \"742530c4-d506-4393-a9e7-709de7b8580e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8dkb"
Aug 04 18:06:16.013466 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:16.013468    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52bpw\" (UniqueName: \"kubernetes.io/projected/742530c4-d506-4393-a9e7-709de7b8580e-kube-api-access-52bpw\") pod \"openshift-lws-operator-fd8ccff4c-q8dkb\" (UID: \"742530c4-d506-4393-a9e7-709de7b8580e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8dkb"
Aug 04 18:06:16.023040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:16.023008    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8dkb"]
Aug 04 18:06:16.114312 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:16.114268    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/742530c4-d506-4393-a9e7-709de7b8580e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q8dkb\" (UID: \"742530c4-d506-4393-a9e7-709de7b8580e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8dkb"
Aug 04 18:06:16.114312 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:16.114316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52bpw\" (UniqueName: \"kubernetes.io/projected/742530c4-d506-4393-a9e7-709de7b8580e-kube-api-access-52bpw\") pod \"openshift-lws-operator-fd8ccff4c-q8dkb\" (UID: \"742530c4-d506-4393-a9e7-709de7b8580e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8dkb"
Aug 04 18:06:16.114663 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:16.114641    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/742530c4-d506-4393-a9e7-709de7b8580e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q8dkb\" (UID: \"742530c4-d506-4393-a9e7-709de7b8580e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8dkb"
Aug 04 18:06:16.128708 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:16.128677    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52bpw\" (UniqueName: \"kubernetes.io/projected/742530c4-d506-4393-a9e7-709de7b8580e-kube-api-access-52bpw\") pod \"openshift-lws-operator-fd8ccff4c-q8dkb\" (UID: \"742530c4-d506-4393-a9e7-709de7b8580e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8dkb"
Aug 04 18:06:16.258165 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:16.258059    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8dkb"
Aug 04 18:06:16.385372 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:16.385336    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8dkb"]
Aug 04 18:06:16.388332 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:06:16.388301    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod742530c4_d506_4393_a9e7_709de7b8580e.slice/crio-83e64c2806d00a370a086907071bdf1ad24c7942f2cc3019f3c5f06b1e30eddc WatchSource:0}: Error finding container 83e64c2806d00a370a086907071bdf1ad24c7942f2cc3019f3c5f06b1e30eddc: Status 404 returned error can't find the container with id 83e64c2806d00a370a086907071bdf1ad24c7942f2cc3019f3c5f06b1e30eddc
Aug 04 18:06:17.173794 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:17.173761    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8dkb" event={"ID":"742530c4-d506-4393-a9e7-709de7b8580e","Type":"ContainerStarted","Data":"83e64c2806d00a370a086907071bdf1ad24c7942f2cc3019f3c5f06b1e30eddc"}
Aug 04 18:06:20.190753 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:20.190719    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8dkb" event={"ID":"742530c4-d506-4393-a9e7-709de7b8580e","Type":"ContainerStarted","Data":"0592d3b85be7662d8786c6776dc2e7a548bd7f8cb90fccf3aa8f09c29212d02b"}
Aug 04 18:06:20.208454 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:20.208410    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8dkb" podStartSLOduration=2.370705977 podStartE2EDuration="5.208394151s" totalImagesPullingTime="2.837688174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:06:20.206898154 +0000 UTC m=+380.182729865" watchObservedRunningTime="2026-08-04 18:06:20.208394151 +0000 UTC m=+380.184225848"
Aug 04 18:06:26.537167 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:26.537132    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:31.100893 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:31.100845    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:31.189351 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:31.189314    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:32.214252 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:32.214215    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:34.543161 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:34.543102    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:34.649591 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:34.649557    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:37.867634 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:37.867602    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:37.961540 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:37.961508    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:40.973251 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:40.973215    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:43.049215 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.049160    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:43.053007 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.052983    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"]
Aug 04 18:06:43.059489 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.059472    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:43.062260 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.062241    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Aug 04 18:06:43.062998 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.062981    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Aug 04 18:06:43.063086 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.062987    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pg58q\""
Aug 04 18:06:43.063086 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.063061    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Aug 04 18:06:43.065920 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.065899    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"]
Aug 04 18:06:43.142599 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.142563    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:43.149647 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.149607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fc7c669-43c4-4ade-8f8f-1c7327792580-metrics-cert\") pod \"lws-controller-manager-68dff9b49f-ztkcf\" (UID: \"7fc7c669-43c4-4ade-8f8f-1c7327792580\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:43.149808 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.149730    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7fc7c669-43c4-4ade-8f8f-1c7327792580-manager-config\") pod \"lws-controller-manager-68dff9b49f-ztkcf\" (UID: \"7fc7c669-43c4-4ade-8f8f-1c7327792580\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:43.149808 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.149783    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hf7h\" (UniqueName: \"kubernetes.io/projected/7fc7c669-43c4-4ade-8f8f-1c7327792580-kube-api-access-5hf7h\") pod \"lws-controller-manager-68dff9b49f-ztkcf\" (UID: \"7fc7c669-43c4-4ade-8f8f-1c7327792580\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:43.149922 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.149862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fc7c669-43c4-4ade-8f8f-1c7327792580-cert\") pod \"lws-controller-manager-68dff9b49f-ztkcf\" (UID: \"7fc7c669-43c4-4ade-8f8f-1c7327792580\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:43.250866 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.250834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fc7c669-43c4-4ade-8f8f-1c7327792580-metrics-cert\") pod \"lws-controller-manager-68dff9b49f-ztkcf\" (UID: \"7fc7c669-43c4-4ade-8f8f-1c7327792580\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:43.251032 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.250903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7fc7c669-43c4-4ade-8f8f-1c7327792580-manager-config\") pod \"lws-controller-manager-68dff9b49f-ztkcf\" (UID: \"7fc7c669-43c4-4ade-8f8f-1c7327792580\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:43.251032 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.250943    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hf7h\" (UniqueName: \"kubernetes.io/projected/7fc7c669-43c4-4ade-8f8f-1c7327792580-kube-api-access-5hf7h\") pod \"lws-controller-manager-68dff9b49f-ztkcf\" (UID: \"7fc7c669-43c4-4ade-8f8f-1c7327792580\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:43.251032 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.250993    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fc7c669-43c4-4ade-8f8f-1c7327792580-cert\") pod \"lws-controller-manager-68dff9b49f-ztkcf\" (UID: \"7fc7c669-43c4-4ade-8f8f-1c7327792580\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:43.251621 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.251596    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7fc7c669-43c4-4ade-8f8f-1c7327792580-manager-config\") pod \"lws-controller-manager-68dff9b49f-ztkcf\" (UID: \"7fc7c669-43c4-4ade-8f8f-1c7327792580\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:43.253446 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.253424    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fc7c669-43c4-4ade-8f8f-1c7327792580-metrics-cert\") pod \"lws-controller-manager-68dff9b49f-ztkcf\" (UID: \"7fc7c669-43c4-4ade-8f8f-1c7327792580\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:43.253446 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.253434    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fc7c669-43c4-4ade-8f8f-1c7327792580-cert\") pod \"lws-controller-manager-68dff9b49f-ztkcf\" (UID: \"7fc7c669-43c4-4ade-8f8f-1c7327792580\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:43.261883 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.261862    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hf7h\" (UniqueName: \"kubernetes.io/projected/7fc7c669-43c4-4ade-8f8f-1c7327792580-kube-api-access-5hf7h\") pod \"lws-controller-manager-68dff9b49f-ztkcf\" (UID: \"7fc7c669-43c4-4ade-8f8f-1c7327792580\") " pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:43.370515 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.370434    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:43.442661 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.442622    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:43.515589 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.515567    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"]
Aug 04 18:06:43.518584 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:06:43.518548    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc7c669_43c4_4ade_8f8f_1c7327792580.slice/crio-71d7b7e59b01a12b96d73f7b5376692aef30d0c5fb2269ae44945af13110916f WatchSource:0}: Error finding container 71d7b7e59b01a12b96d73f7b5376692aef30d0c5fb2269ae44945af13110916f: Status 404 returned error can't find the container with id 71d7b7e59b01a12b96d73f7b5376692aef30d0c5fb2269ae44945af13110916f
Aug 04 18:06:43.527021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:43.526985    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:44.288181 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:44.288150    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf" event={"ID":"7fc7c669-43c4-4ade-8f8f-1c7327792580","Type":"ContainerStarted","Data":"71d7b7e59b01a12b96d73f7b5376692aef30d0c5fb2269ae44945af13110916f"}
Aug 04 18:06:45.295584 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:45.295541    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf" event={"ID":"7fc7c669-43c4-4ade-8f8f-1c7327792580","Type":"ContainerStarted","Data":"24f65ecbc4ddd17058ab58811024d171587a527a7aca48bfa9f224013cc024c7"}
Aug 04 18:06:45.296010 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:45.295666    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:06:45.313568 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:45.313519    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf" podStartSLOduration=0.633696321 podStartE2EDuration="2.313503203s" totalImagesPullingTime="1.679806882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:06:45.312142928 +0000 UTC m=+405.287974648" watchObservedRunningTime="2026-08-04 18:06:45.313503203 +0000 UTC m=+405.289334893"
Aug 04 18:06:56.301586 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:06:56.301557    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68dff9b49f-ztkcf"
Aug 04 18:07:07.277936 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:07.277895    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:08.071646 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:08.071607    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:08.861618 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:08.861586    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:11.917870 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:11.917831    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:12.021919 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:12.021886    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:12.465480 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:12.465449    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:14.045639 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:14.045610    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:14.372992 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:14.372889    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:14.468124 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:14.468085    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:17.068646 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:17.068609    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:17.363269 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:17.363185    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:17.470968 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:17.470919    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:18.719914 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:18.719880    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-jjv5b"]
Aug 04 18:07:18.722593 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:18.722577    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-jjv5b"
Aug 04 18:07:18.725139 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:18.725101    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Aug 04 18:07:18.725253 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:18.725138    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Aug 04 18:07:18.725967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:18.725950    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2jznn\""
Aug 04 18:07:18.734023 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:18.734002    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-jjv5b"]
Aug 04 18:07:18.776522 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:18.776489    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f77n2\" (UniqueName: \"kubernetes.io/projected/f7f3e0c7-3710-4b64-a1a0-d7a40c1f5fdd-kube-api-access-f77n2\") pod \"limitador-operator-controller-manager-774d94476f-jjv5b\" (UID: \"f7f3e0c7-3710-4b64-a1a0-d7a40c1f5fdd\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-jjv5b"
Aug 04 18:07:18.877122 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:18.877091    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f77n2\" (UniqueName: \"kubernetes.io/projected/f7f3e0c7-3710-4b64-a1a0-d7a40c1f5fdd-kube-api-access-f77n2\") pod \"limitador-operator-controller-manager-774d94476f-jjv5b\" (UID: \"f7f3e0c7-3710-4b64-a1a0-d7a40c1f5fdd\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-jjv5b"
Aug 04 18:07:18.886977 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:18.886950    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f77n2\" (UniqueName: \"kubernetes.io/projected/f7f3e0c7-3710-4b64-a1a0-d7a40c1f5fdd-kube-api-access-f77n2\") pod \"limitador-operator-controller-manager-774d94476f-jjv5b\" (UID: \"f7f3e0c7-3710-4b64-a1a0-d7a40c1f5fdd\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-jjv5b"
Aug 04 18:07:19.033359 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:19.033274    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-jjv5b"
Aug 04 18:07:19.099359 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:19.099332    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:19.167445 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:19.167414    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-jjv5b"]
Aug 04 18:07:19.170006 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:07:19.169976    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f3e0c7_3710_4b64_a1a0_d7a40c1f5fdd.slice/crio-87992b04c894bdfd5dbfe78e54ed188b22c95f2545fc718472ebe8c8f6306620 WatchSource:0}: Error finding container 87992b04c894bdfd5dbfe78e54ed188b22c95f2545fc718472ebe8c8f6306620: Status 404 returned error can't find the container with id 87992b04c894bdfd5dbfe78e54ed188b22c95f2545fc718472ebe8c8f6306620
Aug 04 18:07:19.437908 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:19.437868    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:19.439261 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:19.439239    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-jjv5b" event={"ID":"f7f3e0c7-3710-4b64-a1a0-d7a40c1f5fdd","Type":"ContainerStarted","Data":"87992b04c894bdfd5dbfe78e54ed188b22c95f2545fc718472ebe8c8f6306620"}
Aug 04 18:07:19.649131 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:19.649075    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:22.058798 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:22.058756    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:22.159338 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:22.159283    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:22.457366 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:22.457275    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-jjv5b" event={"ID":"f7f3e0c7-3710-4b64-a1a0-d7a40c1f5fdd","Type":"ContainerStarted","Data":"799ed71e1162983eebdfcfc23e95c2fab9fbbdcbc7800b73e5531b966dc1d160"}
Aug 04 18:07:22.457506 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:22.457418    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-jjv5b"
Aug 04 18:07:22.477101 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:22.477051    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-jjv5b" podStartSLOduration=1.898702871 podStartE2EDuration="4.477035971s" totalImagesPullingTime="2.5783331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:07:22.475104726 +0000 UTC m=+442.450936424" watchObservedRunningTime="2026-08-04 18:07:22.477035971 +0000 UTC m=+442.452867668"
Aug 04 18:07:22.663405 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:22.663368    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:22.724447 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:22.724371    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:23.264216 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:23.264187    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:23.331941 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:23.331908    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:29.462073 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:29.462038    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:29.568583 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:29.568553    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:33.463205 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:33.463177    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-jjv5b"
Aug 04 18:07:35.267135 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:35.267074    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:35.368710 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:35.368670    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:38.270677 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:38.270642    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:38.371074 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:38.371037    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:43.159262 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:43.159225    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:43.247127 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:43.247078    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:43.351161 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:43.351127    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:49.643278 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:49.643236    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:49.742643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:49.742615    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:54.394712 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:54.394673    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:54.495451 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:54.495413    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:56.171120 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:56.171084    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:56.271986 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:56.271950    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:59.168957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:59.168921    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:59.276715 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:07:59.276684    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:01.157295 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.157228    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"]
Aug 04 18:08:01.161547 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.161524    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"
Aug 04 18:08:01.165560 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.165535    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:01.165942 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.165922    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Aug 04 18:08:01.166040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.166010    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g7sl8\""
Aug 04 18:08:01.182314 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.182294    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"]
Aug 04 18:08:01.258860 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.258832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzd2g\" (UniqueName: \"kubernetes.io/projected/864beb0b-033f-4b7b-a191-387228148765-kube-api-access-zzd2g\") pod \"limitador-limitador-859c4d96d6-2bqbl\" (UID: \"864beb0b-033f-4b7b-a191-387228148765\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"
Aug 04 18:08:01.259045 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.258880    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/864beb0b-033f-4b7b-a191-387228148765-config-file\") pod \"limitador-limitador-859c4d96d6-2bqbl\" (UID: \"864beb0b-033f-4b7b-a191-387228148765\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"
Aug 04 18:08:01.268001 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.267968    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"]
Aug 04 18:08:01.294456 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.294427    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:01.360197 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.360163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzd2g\" (UniqueName: \"kubernetes.io/projected/864beb0b-033f-4b7b-a191-387228148765-kube-api-access-zzd2g\") pod \"limitador-limitador-859c4d96d6-2bqbl\" (UID: \"864beb0b-033f-4b7b-a191-387228148765\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"
Aug 04 18:08:01.360380 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.360206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/864beb0b-033f-4b7b-a191-387228148765-config-file\") pod \"limitador-limitador-859c4d96d6-2bqbl\" (UID: \"864beb0b-033f-4b7b-a191-387228148765\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"
Aug 04 18:08:01.360775 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.360757    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/864beb0b-033f-4b7b-a191-387228148765-config-file\") pod \"limitador-limitador-859c4d96d6-2bqbl\" (UID: \"864beb0b-033f-4b7b-a191-387228148765\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"
Aug 04 18:08:01.369279 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.369246    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzd2g\" (UniqueName: \"kubernetes.io/projected/864beb0b-033f-4b7b-a191-387228148765-kube-api-access-zzd2g\") pod \"limitador-limitador-859c4d96d6-2bqbl\" (UID: \"864beb0b-033f-4b7b-a191-387228148765\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"
Aug 04 18:08:01.395675 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.395643    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:01.471595 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.471514    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"
Aug 04 18:08:01.614734 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.614707    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"]
Aug 04 18:08:01.616678 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:08:01.616649    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864beb0b_033f_4b7b_a191_387228148765.slice/crio-4a5eaf3271021e6cb3a59891e18e5598c03b9a6fc3a03c51c83b87b583ae529b WatchSource:0}: Error finding container 4a5eaf3271021e6cb3a59891e18e5598c03b9a6fc3a03c51c83b87b583ae529b: Status 404 returned error can't find the container with id 4a5eaf3271021e6cb3a59891e18e5598c03b9a6fc3a03c51c83b87b583ae529b
Aug 04 18:08:01.632525 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:01.632496    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl" event={"ID":"864beb0b-033f-4b7b-a191-387228148765","Type":"ContainerStarted","Data":"4a5eaf3271021e6cb3a59891e18e5598c03b9a6fc3a03c51c83b87b583ae529b"}
Aug 04 18:08:02.146870 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:02.146837    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:02.261184 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:02.261151    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:03.643246 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:03.643205    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl" event={"ID":"864beb0b-033f-4b7b-a191-387228148765","Type":"ContainerStarted","Data":"c15b2bff59c49887efef44303cd58fb94f214eed5bb1f9279b4493c0018b1aa9"}
Aug 04 18:08:03.643648 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:03.643347    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"
Aug 04 18:08:03.664718 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:03.664667    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl" podStartSLOduration=1.479205213 podStartE2EDuration="2.664651781s" totalImagesPullingTime="1.185446568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:08:03.662894773 +0000 UTC m=+483.638726471" watchObservedRunningTime="2026-08-04 18:08:03.664651781 +0000 UTC m=+483.640483477"
Aug 04 18:08:04.379334 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:04.379295    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:04.492243 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:04.492206    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:04.779673 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:04.779591    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:04.879827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:04.879790    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:04.987283 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:04.987252    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:05.069610 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:05.069580    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:11.444221 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:11.444183    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:11.545506 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:11.545472    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:14.648572 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:14.648544    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"
Aug 04 18:08:15.451476 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:15.451438    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"]
Aug 04 18:08:15.451691 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:15.451666    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl" podUID="864beb0b-033f-4b7b-a191-387228148765" containerName="limitador" containerID="cri-o://c15b2bff59c49887efef44303cd58fb94f214eed5bb1f9279b4493c0018b1aa9" gracePeriod=30
Aug 04 18:08:15.994237 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:15.994212    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"
Aug 04 18:08:16.100848 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.100795    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzd2g\" (UniqueName: \"kubernetes.io/projected/864beb0b-033f-4b7b-a191-387228148765-kube-api-access-zzd2g\") pod \"864beb0b-033f-4b7b-a191-387228148765\" (UID: \"864beb0b-033f-4b7b-a191-387228148765\") "
Aug 04 18:08:16.101057 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.100902    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/864beb0b-033f-4b7b-a191-387228148765-config-file\") pod \"864beb0b-033f-4b7b-a191-387228148765\" (UID: \"864beb0b-033f-4b7b-a191-387228148765\") "
Aug 04 18:08:16.101288 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.101265    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864beb0b-033f-4b7b-a191-387228148765-config-file" (OuterVolumeSpecName: "config-file") pod "864beb0b-033f-4b7b-a191-387228148765" (UID: "864beb0b-033f-4b7b-a191-387228148765"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:08:16.102997 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.102980    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864beb0b-033f-4b7b-a191-387228148765-kube-api-access-zzd2g" (OuterVolumeSpecName: "kube-api-access-zzd2g") pod "864beb0b-033f-4b7b-a191-387228148765" (UID: "864beb0b-033f-4b7b-a191-387228148765"). InnerVolumeSpecName "kube-api-access-zzd2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:08:16.201541 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.201494    2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/864beb0b-033f-4b7b-a191-387228148765-config-file\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:08:16.201541 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.201535    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzd2g\" (UniqueName: \"kubernetes.io/projected/864beb0b-033f-4b7b-a191-387228148765-kube-api-access-zzd2g\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:08:16.706658 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.706620    2576 generic.go:356] "Generic (PLEG): container finished" podID="864beb0b-033f-4b7b-a191-387228148765" containerID="c15b2bff59c49887efef44303cd58fb94f214eed5bb1f9279b4493c0018b1aa9" exitCode=0
Aug 04 18:08:16.706853 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.706660    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"
Aug 04 18:08:16.706853 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.706698    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl" event={"ID":"864beb0b-033f-4b7b-a191-387228148765","Type":"ContainerDied","Data":"c15b2bff59c49887efef44303cd58fb94f214eed5bb1f9279b4493c0018b1aa9"}
Aug 04 18:08:16.706853 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.706751    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2bqbl" event={"ID":"864beb0b-033f-4b7b-a191-387228148765","Type":"ContainerDied","Data":"4a5eaf3271021e6cb3a59891e18e5598c03b9a6fc3a03c51c83b87b583ae529b"}
Aug 04 18:08:16.706853 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.706773    2576 scope.go:117] "RemoveContainer" containerID="c15b2bff59c49887efef44303cd58fb94f214eed5bb1f9279b4493c0018b1aa9"
Aug 04 18:08:16.714631 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.714607    2576 scope.go:117] "RemoveContainer" containerID="c15b2bff59c49887efef44303cd58fb94f214eed5bb1f9279b4493c0018b1aa9"
Aug 04 18:08:16.714934 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:08:16.714901    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15b2bff59c49887efef44303cd58fb94f214eed5bb1f9279b4493c0018b1aa9\": container with ID starting with c15b2bff59c49887efef44303cd58fb94f214eed5bb1f9279b4493c0018b1aa9 not found: ID does not exist" containerID="c15b2bff59c49887efef44303cd58fb94f214eed5bb1f9279b4493c0018b1aa9"
Aug 04 18:08:16.715034 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.714954    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15b2bff59c49887efef44303cd58fb94f214eed5bb1f9279b4493c0018b1aa9"} err="failed to get container status \"c15b2bff59c49887efef44303cd58fb94f214eed5bb1f9279b4493c0018b1aa9\": rpc error: code = NotFound desc = could not find container \"c15b2bff59c49887efef44303cd58fb94f214eed5bb1f9279b4493c0018b1aa9\": container with ID starting with c15b2bff59c49887efef44303cd58fb94f214eed5bb1f9279b4493c0018b1aa9 not found: ID does not exist"
Aug 04 18:08:16.725635 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.725608    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"]
Aug 04 18:08:16.730813 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:16.730793    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2bqbl"]
Aug 04 18:08:18.038351 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:18.038320    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:18.129193 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:18.129155    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:18.581966 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:18.581923    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="864beb0b-033f-4b7b-a191-387228148765" path="/var/lib/kubelet/pods/864beb0b-033f-4b7b-a191-387228148765/volumes"
Aug 04 18:08:21.189235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:21.189203    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:21.291002 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:21.290971    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:21.558356 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:21.558328    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:21.716923 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:21.716891    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:24.196836 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.196801    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf4765b85-mm56x"]
Aug 04 18:08:24.197227 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.197171    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="864beb0b-033f-4b7b-a191-387228148765" containerName="limitador"
Aug 04 18:08:24.197227 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.197183    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="864beb0b-033f-4b7b-a191-387228148765" containerName="limitador"
Aug 04 18:08:24.197301 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.197255    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="864beb0b-033f-4b7b-a191-387228148765" containerName="limitador"
Aug 04 18:08:24.200149 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.200132    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-mm56x"
Aug 04 18:08:24.202609 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.202584    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Aug 04 18:08:24.203471 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.203456    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n9vxf\""
Aug 04 18:08:24.207437 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.207421    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-mm56x"]
Aug 04 18:08:24.271750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.271719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg62v\" (UniqueName: \"kubernetes.io/projected/577e1807-3f75-4898-8590-7edc7aca456a-kube-api-access-wg62v\") pod \"authorino-7cf4765b85-mm56x\" (UID: \"577e1807-3f75-4898-8590-7edc7aca456a\") " pod="kuadrant-system/authorino-7cf4765b85-mm56x"
Aug 04 18:08:24.271901 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.271763    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/577e1807-3f75-4898-8590-7edc7aca456a-tls-cert\") pod \"authorino-7cf4765b85-mm56x\" (UID: \"577e1807-3f75-4898-8590-7edc7aca456a\") " pod="kuadrant-system/authorino-7cf4765b85-mm56x"
Aug 04 18:08:24.373227 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.373192    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/577e1807-3f75-4898-8590-7edc7aca456a-tls-cert\") pod \"authorino-7cf4765b85-mm56x\" (UID: \"577e1807-3f75-4898-8590-7edc7aca456a\") " pod="kuadrant-system/authorino-7cf4765b85-mm56x"
Aug 04 18:08:24.373380 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.373283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg62v\" (UniqueName: \"kubernetes.io/projected/577e1807-3f75-4898-8590-7edc7aca456a-kube-api-access-wg62v\") pod \"authorino-7cf4765b85-mm56x\" (UID: \"577e1807-3f75-4898-8590-7edc7aca456a\") " pod="kuadrant-system/authorino-7cf4765b85-mm56x"
Aug 04 18:08:24.375685 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.375663    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/577e1807-3f75-4898-8590-7edc7aca456a-tls-cert\") pod \"authorino-7cf4765b85-mm56x\" (UID: \"577e1807-3f75-4898-8590-7edc7aca456a\") " pod="kuadrant-system/authorino-7cf4765b85-mm56x"
Aug 04 18:08:24.381176 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.381153    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg62v\" (UniqueName: \"kubernetes.io/projected/577e1807-3f75-4898-8590-7edc7aca456a-kube-api-access-wg62v\") pod \"authorino-7cf4765b85-mm56x\" (UID: \"577e1807-3f75-4898-8590-7edc7aca456a\") " pod="kuadrant-system/authorino-7cf4765b85-mm56x"
Aug 04 18:08:24.510570 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.510478    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-mm56x"
Aug 04 18:08:24.638084 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.638056    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-mm56x"]
Aug 04 18:08:24.640277 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:08:24.640247    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod577e1807_3f75_4898_8590_7edc7aca456a.slice/crio-c49d70ec0b07c1e421648c8c29ddd166891e7c538c8f08b8ef42203b45fcef1f WatchSource:0}: Error finding container c49d70ec0b07c1e421648c8c29ddd166891e7c538c8f08b8ef42203b45fcef1f: Status 404 returned error can't find the container with id c49d70ec0b07c1e421648c8c29ddd166891e7c538c8f08b8ef42203b45fcef1f
Aug 04 18:08:24.743060 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:24.743025    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-mm56x" event={"ID":"577e1807-3f75-4898-8590-7edc7aca456a","Type":"ContainerStarted","Data":"c49d70ec0b07c1e421648c8c29ddd166891e7c538c8f08b8ef42203b45fcef1f"}
Aug 04 18:08:26.754569 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:26.754482    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-mm56x" event={"ID":"577e1807-3f75-4898-8590-7edc7aca456a","Type":"ContainerStarted","Data":"5d9a6a20087d6b2c4e266ece03658726c2d8b86f8b1b11f00980336fa1e6e5b5"}
Aug 04 18:08:26.780633 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:26.780584    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf4765b85-mm56x" podStartSLOduration=1.007896861 podStartE2EDuration="2.780568652s" totalImagesPullingTime="1.772671791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:08:26.77948607 +0000 UTC m=+506.755317767" watchObservedRunningTime="2026-08-04 18:08:26.780568652 +0000 UTC m=+506.756400348"
Aug 04 18:08:29.593622 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:29.593588    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:29.715247 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:29.715209    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:55.129764 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.129675    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:55.182018 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.181981    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-5db69799c8-6bpxm"]
Aug 04 18:08:55.184418 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.184397    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-5db69799c8-6bpxm"
Aug 04 18:08:55.186861 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.186837    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Aug 04 18:08:55.187824 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.187808    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Aug 04 18:08:55.187894 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.187822    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Aug 04 18:08:55.187940 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.187892    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-8zf2h\""
Aug 04 18:08:55.194955 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.194921    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-5db69799c8-6bpxm"]
Aug 04 18:08:55.241363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.241327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97ad7257-8e03-4558-bd98-d0a752177a5f-cert\") pod \"llmisvc-controller-manager-5db69799c8-6bpxm\" (UID: \"97ad7257-8e03-4558-bd98-d0a752177a5f\") " pod="kserve/llmisvc-controller-manager-5db69799c8-6bpxm"
Aug 04 18:08:55.241529 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.241401    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvjbj\" (UniqueName: \"kubernetes.io/projected/97ad7257-8e03-4558-bd98-d0a752177a5f-kube-api-access-gvjbj\") pod \"llmisvc-controller-manager-5db69799c8-6bpxm\" (UID: \"97ad7257-8e03-4558-bd98-d0a752177a5f\") " pod="kserve/llmisvc-controller-manager-5db69799c8-6bpxm"
Aug 04 18:08:55.292522 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.292494    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:55.342689 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.342650    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97ad7257-8e03-4558-bd98-d0a752177a5f-cert\") pod \"llmisvc-controller-manager-5db69799c8-6bpxm\" (UID: \"97ad7257-8e03-4558-bd98-d0a752177a5f\") " pod="kserve/llmisvc-controller-manager-5db69799c8-6bpxm"
Aug 04 18:08:55.342869 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.342716    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvjbj\" (UniqueName: \"kubernetes.io/projected/97ad7257-8e03-4558-bd98-d0a752177a5f-kube-api-access-gvjbj\") pod \"llmisvc-controller-manager-5db69799c8-6bpxm\" (UID: \"97ad7257-8e03-4558-bd98-d0a752177a5f\") " pod="kserve/llmisvc-controller-manager-5db69799c8-6bpxm"
Aug 04 18:08:55.345124 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.345096    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97ad7257-8e03-4558-bd98-d0a752177a5f-cert\") pod \"llmisvc-controller-manager-5db69799c8-6bpxm\" (UID: \"97ad7257-8e03-4558-bd98-d0a752177a5f\") " pod="kserve/llmisvc-controller-manager-5db69799c8-6bpxm"
Aug 04 18:08:55.355238 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.355215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvjbj\" (UniqueName: \"kubernetes.io/projected/97ad7257-8e03-4558-bd98-d0a752177a5f-kube-api-access-gvjbj\") pod \"llmisvc-controller-manager-5db69799c8-6bpxm\" (UID: \"97ad7257-8e03-4558-bd98-d0a752177a5f\") " pod="kserve/llmisvc-controller-manager-5db69799c8-6bpxm"
Aug 04 18:08:55.409544 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.409452    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:55.495639 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.495603    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-5db69799c8-6bpxm"
Aug 04 18:08:55.628746 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.628723    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-5db69799c8-6bpxm"]
Aug 04 18:08:55.630931 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:08:55.630898    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod97ad7257_8e03_4558_bd98_d0a752177a5f.slice/crio-f7ad4e0af74649eb3680d7e8f3eb2d13bd549fe8ff40c5d5d5d1e11217bd0022 WatchSource:0}: Error finding container f7ad4e0af74649eb3680d7e8f3eb2d13bd549fe8ff40c5d5d5d1e11217bd0022: Status 404 returned error can't find the container with id f7ad4e0af74649eb3680d7e8f3eb2d13bd549fe8ff40c5d5d5d1e11217bd0022
Aug 04 18:08:55.631429 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.631398    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:55.889192 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:55.889156    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-5db69799c8-6bpxm" event={"ID":"97ad7257-8e03-4558-bd98-d0a752177a5f","Type":"ContainerStarted","Data":"f7ad4e0af74649eb3680d7e8f3eb2d13bd549fe8ff40c5d5d5d1e11217bd0022"}
Aug 04 18:08:56.124060 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:56.124030    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:56.225484 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:56.225391    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:56.333878 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:56.333837    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:59.910198 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:59.910165    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-5db69799c8-6bpxm" event={"ID":"97ad7257-8e03-4558-bd98-d0a752177a5f","Type":"ContainerStarted","Data":"3bdbc722a65715e549774e6c48a88ef4c8d3060134c5dec35afac5f40a9c0a3b"}
Aug 04 18:08:59.910565 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:59.910256    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-5db69799c8-6bpxm"
Aug 04 18:08:59.928124 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:08:59.928075    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-5db69799c8-6bpxm" podStartSLOduration=1.351166334 podStartE2EDuration="4.928060155s" totalImagesPullingTime="3.576893821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:08:59.927162582 +0000 UTC m=+539.902994292" watchObservedRunningTime="2026-08-04 18:08:59.928060155 +0000 UTC m=+539.903891851"
Aug 04 18:09:03.810817 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:09:03.810778    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:16.745188 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:09:16.745146    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:30.917511 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:09:30.917479    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-5db69799c8-6bpxm"
Aug 04 18:09:40.130081 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:09:40.130046    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:41.713100 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:09:41.713061    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:46.789004 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:09:46.788972    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:46.891800 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:09:46.891771    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:52.137799 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:09:52.137739    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:13.609779 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:13.609740    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-wqxwt"]
Aug 04 18:10:13.612300 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:13.612280    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:13.614679 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:13.614659    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Aug 04 18:10:13.614795 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:13.614775    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-jj5mj\""
Aug 04 18:10:13.614850 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:13.614774    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Aug 04 18:10:13.621650 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:13.621626    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-wqxwt"]
Aug 04 18:10:13.649314 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:13.649278    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376dbb1f-8af7-4243-b342-2a20d20418db-metrics-certs\") pod \"odh-model-controller-8588677cfb-wqxwt\" (UID: \"376dbb1f-8af7-4243-b342-2a20d20418db\") " pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:13.649473 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:13.649330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc85q\" (UniqueName: \"kubernetes.io/projected/376dbb1f-8af7-4243-b342-2a20d20418db-kube-api-access-cc85q\") pod \"odh-model-controller-8588677cfb-wqxwt\" (UID: \"376dbb1f-8af7-4243-b342-2a20d20418db\") " pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:13.649473 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:13.649381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/376dbb1f-8af7-4243-b342-2a20d20418db-cert\") pod \"odh-model-controller-8588677cfb-wqxwt\" (UID: \"376dbb1f-8af7-4243-b342-2a20d20418db\") " pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:13.750588 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:13.750553    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/376dbb1f-8af7-4243-b342-2a20d20418db-cert\") pod \"odh-model-controller-8588677cfb-wqxwt\" (UID: \"376dbb1f-8af7-4243-b342-2a20d20418db\") " pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:13.750790 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:13.750617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376dbb1f-8af7-4243-b342-2a20d20418db-metrics-certs\") pod \"odh-model-controller-8588677cfb-wqxwt\" (UID: \"376dbb1f-8af7-4243-b342-2a20d20418db\") " pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:13.750790 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:13.750667    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc85q\" (UniqueName: \"kubernetes.io/projected/376dbb1f-8af7-4243-b342-2a20d20418db-kube-api-access-cc85q\") pod \"odh-model-controller-8588677cfb-wqxwt\" (UID: \"376dbb1f-8af7-4243-b342-2a20d20418db\") " pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:13.750790 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:10:13.750703    2576 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Aug 04 18:10:13.750952 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:10:13.750703    2576 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Aug 04 18:10:13.750952 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:10:13.750800    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376dbb1f-8af7-4243-b342-2a20d20418db-cert podName:376dbb1f-8af7-4243-b342-2a20d20418db nodeName:}" failed. No retries permitted until 2026-08-04 18:10:14.250778811 +0000 UTC m=+614.226610500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/376dbb1f-8af7-4243-b342-2a20d20418db-cert") pod "odh-model-controller-8588677cfb-wqxwt" (UID: "376dbb1f-8af7-4243-b342-2a20d20418db") : secret "odh-model-controller-webhook-cert" not found
Aug 04 18:10:13.750952 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:10:13.750857    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376dbb1f-8af7-4243-b342-2a20d20418db-metrics-certs podName:376dbb1f-8af7-4243-b342-2a20d20418db nodeName:}" failed. No retries permitted until 2026-08-04 18:10:14.2508386 +0000 UTC m=+614.226670286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/376dbb1f-8af7-4243-b342-2a20d20418db-metrics-certs") pod "odh-model-controller-8588677cfb-wqxwt" (UID: "376dbb1f-8af7-4243-b342-2a20d20418db") : secret "odh-model-controller-metrics-tls" not found
Aug 04 18:10:13.759884 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:13.759854    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc85q\" (UniqueName: \"kubernetes.io/projected/376dbb1f-8af7-4243-b342-2a20d20418db-kube-api-access-cc85q\") pod \"odh-model-controller-8588677cfb-wqxwt\" (UID: \"376dbb1f-8af7-4243-b342-2a20d20418db\") " pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:13.978890 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:13.978803    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:14.043519 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:14.043482    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:14.209951 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:14.209919    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:14.255956 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:14.255868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376dbb1f-8af7-4243-b342-2a20d20418db-metrics-certs\") pod \"odh-model-controller-8588677cfb-wqxwt\" (UID: \"376dbb1f-8af7-4243-b342-2a20d20418db\") " pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:14.256144 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:14.255964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/376dbb1f-8af7-4243-b342-2a20d20418db-cert\") pod \"odh-model-controller-8588677cfb-wqxwt\" (UID: \"376dbb1f-8af7-4243-b342-2a20d20418db\") " pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:14.258296 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:14.258271    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376dbb1f-8af7-4243-b342-2a20d20418db-metrics-certs\") pod \"odh-model-controller-8588677cfb-wqxwt\" (UID: \"376dbb1f-8af7-4243-b342-2a20d20418db\") " pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:14.258435 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:14.258308    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/376dbb1f-8af7-4243-b342-2a20d20418db-cert\") pod \"odh-model-controller-8588677cfb-wqxwt\" (UID: \"376dbb1f-8af7-4243-b342-2a20d20418db\") " pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:14.309945 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:14.309913    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:14.523342 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:14.523254    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:14.649392 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:14.649263    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-wqxwt"]
Aug 04 18:10:14.651648 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:10:14.651619    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376dbb1f_8af7_4243_b342_2a20d20418db.slice/crio-4f0d3c897fa453a7c58f5f1fe962304dffcbc55598dfc76709eebcb8959bbbb5 WatchSource:0}: Error finding container 4f0d3c897fa453a7c58f5f1fe962304dffcbc55598dfc76709eebcb8959bbbb5: Status 404 returned error can't find the container with id 4f0d3c897fa453a7c58f5f1fe962304dffcbc55598dfc76709eebcb8959bbbb5
Aug 04 18:10:15.254000 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:15.253970    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-wqxwt" event={"ID":"376dbb1f-8af7-4243-b342-2a20d20418db","Type":"ContainerStarted","Data":"4f0d3c897fa453a7c58f5f1fe962304dffcbc55598dfc76709eebcb8959bbbb5"}
Aug 04 18:10:18.272396 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:18.272361    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-wqxwt" event={"ID":"376dbb1f-8af7-4243-b342-2a20d20418db","Type":"ContainerStarted","Data":"d04940eb8e7dcc5997895e521bc753a7dc795b99154f976051406f4666d8d92a"}
Aug 04 18:10:18.272773 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:18.272504    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:18.292677 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:18.292632    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-wqxwt" podStartSLOduration=2.222083867 podStartE2EDuration="5.292617538s" totalImagesPullingTime="3.070533671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:18.291016468 +0000 UTC m=+618.266848167" watchObservedRunningTime="2026-08-04 18:10:18.292617538 +0000 UTC m=+618.268449235"
Aug 04 18:10:29.279972 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:29.279897    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-wqxwt"
Aug 04 18:10:50.024092 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:50.024059    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:51.988459 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:10:51.988427    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:16.686973 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:11:16.686932    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:16.748626 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:11:16.748596    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:47.569150 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:11:47.569105    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:47.669266 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:11:47.669229    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:00.549341 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:00.549251    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:00.649699 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:00.649662    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:07.953323 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:07.953287    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:08.265039 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.264957    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:08.355709 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.355675    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:08.491473 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.491438    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"]
Aug 04 18:12:08.495020 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.494994    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"
Aug 04 18:12:08.498844 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.498814    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Aug 04 18:12:08.498989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.498883    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-vr8jv\""
Aug 04 18:12:08.498989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.498911    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Aug 04 18:12:08.498989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.498887    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-w8j42\""
Aug 04 18:12:08.498989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.498973    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Aug 04 18:12:08.508712 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.508690    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"]
Aug 04 18:12:08.528030 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.527953    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:08.557020 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.556984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0f28dd-3d13-4ae8-9039-69ed51b761b0-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk\" (UID: \"0b0f28dd-3d13-4ae8-9039-69ed51b761b0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"
Aug 04 18:12:08.557212 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.557086    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nljxg\" (UniqueName: \"kubernetes.io/projected/0b0f28dd-3d13-4ae8-9039-69ed51b761b0-kube-api-access-nljxg\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk\" (UID: \"0b0f28dd-3d13-4ae8-9039-69ed51b761b0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"
Aug 04 18:12:08.657699 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.657658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nljxg\" (UniqueName: \"kubernetes.io/projected/0b0f28dd-3d13-4ae8-9039-69ed51b761b0-kube-api-access-nljxg\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk\" (UID: \"0b0f28dd-3d13-4ae8-9039-69ed51b761b0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"
Aug 04 18:12:08.657869 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.657722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0f28dd-3d13-4ae8-9039-69ed51b761b0-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk\" (UID: \"0b0f28dd-3d13-4ae8-9039-69ed51b761b0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"
Aug 04 18:12:08.660141 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.660097    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0f28dd-3d13-4ae8-9039-69ed51b761b0-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk\" (UID: \"0b0f28dd-3d13-4ae8-9039-69ed51b761b0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"
Aug 04 18:12:08.670100 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.670067    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nljxg\" (UniqueName: \"kubernetes.io/projected/0b0f28dd-3d13-4ae8-9039-69ed51b761b0-kube-api-access-nljxg\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk\" (UID: \"0b0f28dd-3d13-4ae8-9039-69ed51b761b0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"
Aug 04 18:12:08.805462 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.805422    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"
Aug 04 18:12:08.954436 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.954350    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"]
Aug 04 18:12:08.956908 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:12:08.956863    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0f28dd_3d13_4ae8_9039_69ed51b761b0.slice/crio-7678ef4a54b897dab42ef7cf80e01a46db4ac3c206b7f6af697ba85655970c24 WatchSource:0}: Error finding container 7678ef4a54b897dab42ef7cf80e01a46db4ac3c206b7f6af697ba85655970c24: Status 404 returned error can't find the container with id 7678ef4a54b897dab42ef7cf80e01a46db4ac3c206b7f6af697ba85655970c24
Aug 04 18:12:08.960672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:08.960643    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:12:09.788788 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:09.788746    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk" event={"ID":"0b0f28dd-3d13-4ae8-9039-69ed51b761b0","Type":"ContainerStarted","Data":"7678ef4a54b897dab42ef7cf80e01a46db4ac3c206b7f6af697ba85655970c24"}
Aug 04 18:12:11.801472 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:11.801439    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk" event={"ID":"0b0f28dd-3d13-4ae8-9039-69ed51b761b0","Type":"ContainerStarted","Data":"4e97b7bb0d580228cb00b40fb77a45bcf0923fba9b4de491fa392bfd9444f984"}
Aug 04 18:12:11.801840 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:11.801506    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"
Aug 04 18:12:24.097133 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.097056    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk" podStartSLOduration=13.639478313 podStartE2EDuration="16.097037168s" totalImagesPullingTime="2.457558855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:11.830809439 +0000 UTC m=+731.806641137" watchObservedRunningTime="2026-08-04 18:12:24.097037168 +0000 UTC m=+744.072868864"
Aug 04 18:12:24.097791 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.097774    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"]
Aug 04 18:12:24.101505 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.101485    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.105139 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.105102    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Aug 04 18:12:24.105444 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.105123    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Aug 04 18:12:24.105566 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.105123    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Aug 04 18:12:24.105619 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.105167    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-wbcwg\""
Aug 04 18:12:24.110600 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.110576    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"]
Aug 04 18:12:24.135518 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.135490    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:24.203666 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.203631    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-dshm\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.203666 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.203675    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.203930 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.203698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-home\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.203930 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.203807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c055939c-83ef-4b2a-9550-bb027b255a1d-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.203930 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.203845    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxn97\" (UniqueName: \"kubernetes.io/projected/c055939c-83ef-4b2a-9550-bb027b255a1d-kube-api-access-xxn97\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.203930 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.203889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.305206 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.305176    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c055939c-83ef-4b2a-9550-bb027b255a1d-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.305206 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.305210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxn97\" (UniqueName: \"kubernetes.io/projected/c055939c-83ef-4b2a-9550-bb027b255a1d-kube-api-access-xxn97\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.305443 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.305240    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.305443 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.305332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-dshm\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.305443 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.305374    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.305443 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.305399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-home\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.305747 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.305726    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.305957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.305931    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.306153 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.306094    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-home\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.307628 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.307606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-dshm\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.307839 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.307808    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c055939c-83ef-4b2a-9550-bb027b255a1d-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.312397 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.312377    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxn97\" (UniqueName: \"kubernetes.io/projected/c055939c-83ef-4b2a-9550-bb027b255a1d-kube-api-access-xxn97\") pod \"fc-smoke-test-kserve-7f4687897-7ctn9\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.327090 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.327063    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:24.412530 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.412445    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:24.552297 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.552270    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"]
Aug 04 18:12:24.553912 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:12:24.553886    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc055939c_83ef_4b2a_9550_bb027b255a1d.slice/crio-d73fc5e8901c87c2b437ac121c9a2d171a8930172a374031e1a228a5bc96db4b WatchSource:0}: Error finding container d73fc5e8901c87c2b437ac121c9a2d171a8930172a374031e1a228a5bc96db4b: Status 404 returned error can't find the container with id d73fc5e8901c87c2b437ac121c9a2d171a8930172a374031e1a228a5bc96db4b
Aug 04 18:12:24.864171 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:24.864104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9" event={"ID":"c055939c-83ef-4b2a-9550-bb027b255a1d","Type":"ContainerStarted","Data":"d73fc5e8901c87c2b437ac121c9a2d171a8930172a374031e1a228a5bc96db4b"}
Aug 04 18:12:26.879234 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:26.879199    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9" event={"ID":"c055939c-83ef-4b2a-9550-bb027b255a1d","Type":"ContainerStarted","Data":"b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46"}
Aug 04 18:12:26.896271 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:26.896217    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9" podStartSLOduration=1.390874248 podStartE2EDuration="2.896199344s" totalImagesPullingTime="1.505325096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:26.895406291 +0000 UTC m=+746.871237989" watchObservedRunningTime="2026-08-04 18:12:26.896199344 +0000 UTC m=+746.872031041"
Aug 04 18:12:27.413449 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:27.413407    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:34.413612 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:34.413571    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:34.427033 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:34.426997    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:34.933857 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:34.933824    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:12:42.432019 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:42.431981    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:42.808483 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:42.808448    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"
Aug 04 18:12:58.381055 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:12:58.381024    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:20.642272 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:20.642233    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"]
Aug 04 18:13:20.643045 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:20.643008    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:20.653836 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:13:20.653798    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:13:20.653836 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:13:20.653836 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:13:20.653836 ip-10-0-133-187 kubenswrapper[2576]:  > containerID="b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46" cmd=["/bin/sleep","15"]
Aug 04 18:13:20.654129 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:20.653860    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9" message="" err=<
Aug 04 18:13:20.654129 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:13:20.654129 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:13:20.654129 ip-10-0-133-187 kubenswrapper[2576]:  >
Aug 04 18:13:20.654129 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:13:20.653876    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:13:20.654129 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:13:20.654129 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:13:20.654129 ip-10-0-133-187 kubenswrapper[2576]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9" podUID="c055939c-83ef-4b2a-9550-bb027b255a1d" containerName="main" containerID="cri-o://b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46"
Aug 04 18:13:20.654129 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:20.653928    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9" podUID="c055939c-83ef-4b2a-9550-bb027b255a1d" containerName="main" containerID="cri-o://b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46" gracePeriod=60
Aug 04 18:13:20.737800 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:20.737766    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:21.005395 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.005371    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:13:21.035811 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.035776    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-model-cache\") pod \"c055939c-83ef-4b2a-9550-bb027b255a1d\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") "
Aug 04 18:13:21.036000 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.035842    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c055939c-83ef-4b2a-9550-bb027b255a1d-tls-certs\") pod \"c055939c-83ef-4b2a-9550-bb027b255a1d\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") "
Aug 04 18:13:21.036000 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.035883    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-tmp-dir\") pod \"c055939c-83ef-4b2a-9550-bb027b255a1d\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") "
Aug 04 18:13:21.036000 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.035987    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-home\") pod \"c055939c-83ef-4b2a-9550-bb027b255a1d\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") "
Aug 04 18:13:21.036186 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.036020    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxn97\" (UniqueName: \"kubernetes.io/projected/c055939c-83ef-4b2a-9550-bb027b255a1d-kube-api-access-xxn97\") pod \"c055939c-83ef-4b2a-9550-bb027b255a1d\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") "
Aug 04 18:13:21.036186 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.036021    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-model-cache" (OuterVolumeSpecName: "model-cache") pod "c055939c-83ef-4b2a-9550-bb027b255a1d" (UID: "c055939c-83ef-4b2a-9550-bb027b255a1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:13:21.036186 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.036047    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-dshm\") pod \"c055939c-83ef-4b2a-9550-bb027b255a1d\" (UID: \"c055939c-83ef-4b2a-9550-bb027b255a1d\") "
Aug 04 18:13:21.036353 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.036207    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c055939c-83ef-4b2a-9550-bb027b255a1d" (UID: "c055939c-83ef-4b2a-9550-bb027b255a1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:13:21.036353 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.036253    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-home" (OuterVolumeSpecName: "home") pod "c055939c-83ef-4b2a-9550-bb027b255a1d" (UID: "c055939c-83ef-4b2a-9550-bb027b255a1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:13:21.036468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.036397    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:13:21.036468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.036415    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:13:21.036468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.036453    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:13:21.038342 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.038312    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c055939c-83ef-4b2a-9550-bb027b255a1d-kube-api-access-xxn97" (OuterVolumeSpecName: "kube-api-access-xxn97") pod "c055939c-83ef-4b2a-9550-bb027b255a1d" (UID: "c055939c-83ef-4b2a-9550-bb027b255a1d"). InnerVolumeSpecName "kube-api-access-xxn97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:13:21.038471 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.038342    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c055939c-83ef-4b2a-9550-bb027b255a1d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c055939c-83ef-4b2a-9550-bb027b255a1d" (UID: "c055939c-83ef-4b2a-9550-bb027b255a1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:13:21.038471 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.038387    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-dshm" (OuterVolumeSpecName: "dshm") pod "c055939c-83ef-4b2a-9550-bb027b255a1d" (UID: "c055939c-83ef-4b2a-9550-bb027b255a1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:13:21.137040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.137011    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxn97\" (UniqueName: \"kubernetes.io/projected/c055939c-83ef-4b2a-9550-bb027b255a1d-kube-api-access-xxn97\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:13:21.137040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.137037    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c055939c-83ef-4b2a-9550-bb027b255a1d-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:13:21.137040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.137046    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c055939c-83ef-4b2a-9550-bb027b255a1d-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:13:21.144028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.143999    2576 generic.go:356] "Generic (PLEG): container finished" podID="c055939c-83ef-4b2a-9550-bb027b255a1d" containerID="b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46" exitCode=0
Aug 04 18:13:21.144197 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.144045    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"
Aug 04 18:13:21.144197 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.144062    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9" event={"ID":"c055939c-83ef-4b2a-9550-bb027b255a1d","Type":"ContainerDied","Data":"b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46"}
Aug 04 18:13:21.144197 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.144104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9" event={"ID":"c055939c-83ef-4b2a-9550-bb027b255a1d","Type":"ContainerDied","Data":"d73fc5e8901c87c2b437ac121c9a2d171a8930172a374031e1a228a5bc96db4b"}
Aug 04 18:13:21.144197 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.144141    2576 scope.go:117] "RemoveContainer" containerID="b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46"
Aug 04 18:13:21.153998 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.153979    2576 scope.go:117] "RemoveContainer" containerID="b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46"
Aug 04 18:13:21.154290 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:13:21.154274    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46\": container with ID starting with b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46 not found: ID does not exist" containerID="b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46"
Aug 04 18:13:21.154359 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.154298    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46"} err="failed to get container status \"b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46\": rpc error: code = NotFound desc = could not find container \"b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46\": container with ID starting with b82a41aa299fd9b7cdfaa0c1ad3357310c7dae2fc839261b488bf6414e48fe46 not found: ID does not exist"
Aug 04 18:13:21.166029 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.166004    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"]
Aug 04 18:13:21.171790 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.171730    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9"]
Aug 04 18:13:21.923985 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:21.923933    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7ctn9" podUID="c055939c-83ef-4b2a-9550-bb027b255a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Aug 04 18:13:22.583368 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:22.583338    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c055939c-83ef-4b2a-9550-bb027b255a1d" path="/var/lib/kubelet/pods/c055939c-83ef-4b2a-9550-bb027b255a1d/volumes"
Aug 04 18:13:40.676010 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:40.675974    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:40.774423 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:40.774387    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:46.790323 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:46.790270    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:52.358618 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:52.358578    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:52.460255 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:13:52.460221    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:00.403280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.403246    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"]
Aug 04 18:14:00.403643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.403622    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c055939c-83ef-4b2a-9550-bb027b255a1d" containerName="main"
Aug 04 18:14:00.403643 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.403633    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c055939c-83ef-4b2a-9550-bb027b255a1d" containerName="main"
Aug 04 18:14:00.403714 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.403703    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c055939c-83ef-4b2a-9550-bb027b255a1d" containerName="main"
Aug 04 18:14:00.406613 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.406593    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.410693 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.410671    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Aug 04 18:14:00.410693 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.410683    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Aug 04 18:14:00.410882 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.410674    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Aug 04 18:14:00.410882 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.410678    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-mz8cv\""
Aug 04 18:14:00.415158 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.415138    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"]
Aug 04 18:14:00.433055 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.433016    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:00.583625 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.583593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-home\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.583794 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.583670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twd29\" (UniqueName: \"kubernetes.io/projected/ee39a075-ab6f-4fee-a003-f119676501d1-kube-api-access-twd29\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.583794 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.583702    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.583794 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.583753    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.583946 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.583803    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee39a075-ab6f-4fee-a003-f119676501d1-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.583946 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.583836    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.606440 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.606409    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"]
Aug 04 18:14:00.610693 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.610673    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"
Aug 04 18:14:00.613230 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.613208    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-49g8q\""
Aug 04 18:14:00.619687 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.619663    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"]
Aug 04 18:14:00.635762 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.635724    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:00.684379 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.684298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-home\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.684379 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.684357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twd29\" (UniqueName: \"kubernetes.io/projected/ee39a075-ab6f-4fee-a003-f119676501d1-kube-api-access-twd29\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.684587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.684381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.684587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.684422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.684587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.684445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee39a075-ab6f-4fee-a003-f119676501d1-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.684587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.684498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.684749 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.684705    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-home\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.684792 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.684771    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.684857 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.684835    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.688990 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.688969    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee39a075-ab6f-4fee-a003-f119676501d1-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.689162 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.688967    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.692802 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.692777    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twd29\" (UniqueName: \"kubernetes.io/projected/ee39a075-ab6f-4fee-a003-f119676501d1-kube-api-access-twd29\") pod \"fc-concurrency-test-kserve-b5d44467c-rp9kk\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.717073 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.717046    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:00.785859 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.785824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgf5z\" (UniqueName: \"kubernetes.io/projected/0fa25431-48ac-480e-ac57-2ebb23a715ae-kube-api-access-zgf5z\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh\" (UID: \"0fa25431-48ac-480e-ac57-2ebb23a715ae\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"
Aug 04 18:14:00.786005 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.785949    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fa25431-48ac-480e-ac57-2ebb23a715ae-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh\" (UID: \"0fa25431-48ac-480e-ac57-2ebb23a715ae\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"
Aug 04 18:14:00.849321 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.849295    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"]
Aug 04 18:14:00.851605 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:14:00.851577    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee39a075_ab6f_4fee_a003_f119676501d1.slice/crio-9ee4f6959dcad475405778fafc911a7e7da42479f4c43b4c8a11528e6a7f40f9 WatchSource:0}: Error finding container 9ee4f6959dcad475405778fafc911a7e7da42479f4c43b4c8a11528e6a7f40f9: Status 404 returned error can't find the container with id 9ee4f6959dcad475405778fafc911a7e7da42479f4c43b4c8a11528e6a7f40f9
Aug 04 18:14:00.886729 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.886698    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fa25431-48ac-480e-ac57-2ebb23a715ae-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh\" (UID: \"0fa25431-48ac-480e-ac57-2ebb23a715ae\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"
Aug 04 18:14:00.886855 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.886789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgf5z\" (UniqueName: \"kubernetes.io/projected/0fa25431-48ac-480e-ac57-2ebb23a715ae-kube-api-access-zgf5z\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh\" (UID: \"0fa25431-48ac-480e-ac57-2ebb23a715ae\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"
Aug 04 18:14:00.888999 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.888980    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fa25431-48ac-480e-ac57-2ebb23a715ae-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh\" (UID: \"0fa25431-48ac-480e-ac57-2ebb23a715ae\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"
Aug 04 18:14:00.894468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.894446    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgf5z\" (UniqueName: \"kubernetes.io/projected/0fa25431-48ac-480e-ac57-2ebb23a715ae-kube-api-access-zgf5z\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh\" (UID: \"0fa25431-48ac-480e-ac57-2ebb23a715ae\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"
Aug 04 18:14:00.921804 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:00.921774    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"
Aug 04 18:14:01.057904 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:01.057853    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"]
Aug 04 18:14:01.061592 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:14:01.061568    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa25431_48ac_480e_ac57_2ebb23a715ae.slice/crio-95bf077adb679cb4414e777f8c255938531bf8e7c3f083c3552c15836458ad78 WatchSource:0}: Error finding container 95bf077adb679cb4414e777f8c255938531bf8e7c3f083c3552c15836458ad78: Status 404 returned error can't find the container with id 95bf077adb679cb4414e777f8c255938531bf8e7c3f083c3552c15836458ad78
Aug 04 18:14:01.341024 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:01.340983    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh" event={"ID":"0fa25431-48ac-480e-ac57-2ebb23a715ae","Type":"ContainerStarted","Data":"cd95ea171bc37087893dd590d2ddd3951371eda24bc010404769cd6c18dd7527"}
Aug 04 18:14:01.341245 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:01.341049    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh" event={"ID":"0fa25431-48ac-480e-ac57-2ebb23a715ae","Type":"ContainerStarted","Data":"95bf077adb679cb4414e777f8c255938531bf8e7c3f083c3552c15836458ad78"}
Aug 04 18:14:01.341245 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:01.341077    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"
Aug 04 18:14:01.343027 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:01.342997    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk" event={"ID":"ee39a075-ab6f-4fee-a003-f119676501d1","Type":"ContainerStarted","Data":"674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9"}
Aug 04 18:14:01.343180 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:01.343049    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk" event={"ID":"ee39a075-ab6f-4fee-a003-f119676501d1","Type":"ContainerStarted","Data":"9ee4f6959dcad475405778fafc911a7e7da42479f4c43b4c8a11528e6a7f40f9"}
Aug 04 18:14:01.358522 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:01.358454    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh" podStartSLOduration=1.358435861 podStartE2EDuration="1.358435861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:14:01.357693445 +0000 UTC m=+841.333525140" watchObservedRunningTime="2026-08-04 18:14:01.358435861 +0000 UTC m=+841.334267560"
Aug 04 18:14:01.382167 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:01.382089    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk" podStartSLOduration=1.382069818 podStartE2EDuration="1.382069818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:14:01.381460255 +0000 UTC m=+841.357291953" watchObservedRunningTime="2026-08-04 18:14:01.382069818 +0000 UTC m=+841.357901516"
Aug 04 18:14:01.717872 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:01.717804    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:06.234615 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:06.234581    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:10.717408 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:10.717379    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:10.729661 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:10.729632    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:11.401057 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:11.400999    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:23.925179 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:23.925141    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:23.952835 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:23.952793    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"]
Aug 04 18:14:24.016145 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:24.016094    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:32.348985 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:32.348954    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"
Aug 04 18:14:32.808102 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:32.808066    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk" podUID="0b0f28dd-3d13-4ae8-9039-69ed51b761b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:14:38.543155 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:38.541340    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"]
Aug 04 18:14:38.545311 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:38.545282    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"]
Aug 04 18:14:38.555778 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:38.555750    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:38.556334 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:14:38.556296    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:14:38.556334 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:14:38.556334 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:14:38.556334 ip-10-0-133-187 kubenswrapper[2576]:  > containerID="674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9" cmd=["/bin/sleep","15"]
Aug 04 18:14:38.556567 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:38.556355    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk" message="" err=<
Aug 04 18:14:38.556567 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:14:38.556567 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:14:38.556567 ip-10-0-133-187 kubenswrapper[2576]:  >
Aug 04 18:14:38.556567 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:14:38.556371    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:14:38.556567 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:14:38.556567 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:14:38.556567 ip-10-0-133-187 kubenswrapper[2576]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk" podUID="ee39a075-ab6f-4fee-a003-f119676501d1" containerName="main" containerID="cri-o://674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9"
Aug 04 18:14:38.556567 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:38.556426    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk" podUID="ee39a075-ab6f-4fee-a003-f119676501d1" containerName="main" containerID="cri-o://674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9" gracePeriod=60
Aug 04 18:14:38.619095 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:38.619060    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:38.905891 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:38.905868    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:38.974588 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:38.974555    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk" podUID="0b0f28dd-3d13-4ae8-9039-69ed51b761b0" containerName="main" containerID="cri-o://4e97b7bb0d580228cb00b40fb77a45bcf0923fba9b4de491fa392bfd9444f984" gracePeriod=45
Aug 04 18:14:39.040738 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.040709    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-home\") pod \"ee39a075-ab6f-4fee-a003-f119676501d1\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") "
Aug 04 18:14:39.040904 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.040774    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twd29\" (UniqueName: \"kubernetes.io/projected/ee39a075-ab6f-4fee-a003-f119676501d1-kube-api-access-twd29\") pod \"ee39a075-ab6f-4fee-a003-f119676501d1\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") "
Aug 04 18:14:39.040904 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.040812    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-dshm\") pod \"ee39a075-ab6f-4fee-a003-f119676501d1\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") "
Aug 04 18:14:39.040904 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.040855    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-model-cache\") pod \"ee39a075-ab6f-4fee-a003-f119676501d1\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") "
Aug 04 18:14:39.041102 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.040908    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee39a075-ab6f-4fee-a003-f119676501d1-tls-certs\") pod \"ee39a075-ab6f-4fee-a003-f119676501d1\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") "
Aug 04 18:14:39.041102 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.040938    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-tmp-dir\") pod \"ee39a075-ab6f-4fee-a003-f119676501d1\" (UID: \"ee39a075-ab6f-4fee-a003-f119676501d1\") "
Aug 04 18:14:39.041102 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.040989    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-home" (OuterVolumeSpecName: "home") pod "ee39a075-ab6f-4fee-a003-f119676501d1" (UID: "ee39a075-ab6f-4fee-a003-f119676501d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:39.041280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.041206    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee39a075-ab6f-4fee-a003-f119676501d1" (UID: "ee39a075-ab6f-4fee-a003-f119676501d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:39.041280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.041250    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.041357 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.041313    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee39a075-ab6f-4fee-a003-f119676501d1" (UID: "ee39a075-ab6f-4fee-a003-f119676501d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:39.043080 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.043051    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-dshm" (OuterVolumeSpecName: "dshm") pod "ee39a075-ab6f-4fee-a003-f119676501d1" (UID: "ee39a075-ab6f-4fee-a003-f119676501d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:39.043080 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.043060    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee39a075-ab6f-4fee-a003-f119676501d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee39a075-ab6f-4fee-a003-f119676501d1" (UID: "ee39a075-ab6f-4fee-a003-f119676501d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:14:39.043248 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.043140    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee39a075-ab6f-4fee-a003-f119676501d1-kube-api-access-twd29" (OuterVolumeSpecName: "kube-api-access-twd29") pod "ee39a075-ab6f-4fee-a003-f119676501d1" (UID: "ee39a075-ab6f-4fee-a003-f119676501d1"). InnerVolumeSpecName "kube-api-access-twd29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:14:39.142442 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.142412    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twd29\" (UniqueName: \"kubernetes.io/projected/ee39a075-ab6f-4fee-a003-f119676501d1-kube-api-access-twd29\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.142442 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.142443    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.142623 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.142458    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.142623 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.142492    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee39a075-ab6f-4fee-a003-f119676501d1-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.142623 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.142501    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee39a075-ab6f-4fee-a003-f119676501d1-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.217452 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.217424    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"
Aug 04 18:14:39.344266 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.344175    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nljxg\" (UniqueName: \"kubernetes.io/projected/0b0f28dd-3d13-4ae8-9039-69ed51b761b0-kube-api-access-nljxg\") pod \"0b0f28dd-3d13-4ae8-9039-69ed51b761b0\" (UID: \"0b0f28dd-3d13-4ae8-9039-69ed51b761b0\") "
Aug 04 18:14:39.344266 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.344253    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0f28dd-3d13-4ae8-9039-69ed51b761b0-tls-certs\") pod \"0b0f28dd-3d13-4ae8-9039-69ed51b761b0\" (UID: \"0b0f28dd-3d13-4ae8-9039-69ed51b761b0\") "
Aug 04 18:14:39.346432 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.346401    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0f28dd-3d13-4ae8-9039-69ed51b761b0-kube-api-access-nljxg" (OuterVolumeSpecName: "kube-api-access-nljxg") pod "0b0f28dd-3d13-4ae8-9039-69ed51b761b0" (UID: "0b0f28dd-3d13-4ae8-9039-69ed51b761b0"). InnerVolumeSpecName "kube-api-access-nljxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:14:39.346432 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.346410    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0f28dd-3d13-4ae8-9039-69ed51b761b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b0f28dd-3d13-4ae8-9039-69ed51b761b0" (UID: "0b0f28dd-3d13-4ae8-9039-69ed51b761b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:14:39.445481 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.445437    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0f28dd-3d13-4ae8-9039-69ed51b761b0-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.445481 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.445474    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nljxg\" (UniqueName: \"kubernetes.io/projected/0b0f28dd-3d13-4ae8-9039-69ed51b761b0-kube-api-access-nljxg\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:14:39.528170 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.528134    2576 generic.go:356] "Generic (PLEG): container finished" podID="ee39a075-ab6f-4fee-a003-f119676501d1" containerID="674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9" exitCode=0
Aug 04 18:14:39.528356 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.528230    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk" event={"ID":"ee39a075-ab6f-4fee-a003-f119676501d1","Type":"ContainerDied","Data":"674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9"}
Aug 04 18:14:39.528356 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.528259    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"
Aug 04 18:14:39.528356 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.528272    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk" event={"ID":"ee39a075-ab6f-4fee-a003-f119676501d1","Type":"ContainerDied","Data":"9ee4f6959dcad475405778fafc911a7e7da42479f4c43b4c8a11528e6a7f40f9"}
Aug 04 18:14:39.528356 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.528289    2576 scope.go:117] "RemoveContainer" containerID="674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9"
Aug 04 18:14:39.529528 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.529507    2576 generic.go:356] "Generic (PLEG): container finished" podID="0b0f28dd-3d13-4ae8-9039-69ed51b761b0" containerID="4e97b7bb0d580228cb00b40fb77a45bcf0923fba9b4de491fa392bfd9444f984" exitCode=0
Aug 04 18:14:39.529621 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.529556    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"
Aug 04 18:14:39.529621 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.529593    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk" event={"ID":"0b0f28dd-3d13-4ae8-9039-69ed51b761b0","Type":"ContainerDied","Data":"4e97b7bb0d580228cb00b40fb77a45bcf0923fba9b4de491fa392bfd9444f984"}
Aug 04 18:14:39.529749 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.529623    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk" event={"ID":"0b0f28dd-3d13-4ae8-9039-69ed51b761b0","Type":"ContainerDied","Data":"7678ef4a54b897dab42ef7cf80e01a46db4ac3c206b7f6af697ba85655970c24"}
Aug 04 18:14:39.538651 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.538636    2576 scope.go:117] "RemoveContainer" containerID="674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9"
Aug 04 18:14:39.538922 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:14:39.538902    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9\": container with ID starting with 674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9 not found: ID does not exist" containerID="674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9"
Aug 04 18:14:39.539001 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.538935    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9"} err="failed to get container status \"674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9\": rpc error: code = NotFound desc = could not find container \"674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9\": container with ID starting with 674fa364ff154dfe630ac680606902d59ec432c35cb6e288fdbeb91bb8b59af9 not found: ID does not exist"
Aug 04 18:14:39.539001 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.538958    2576 scope.go:117] "RemoveContainer" containerID="4e97b7bb0d580228cb00b40fb77a45bcf0923fba9b4de491fa392bfd9444f984"
Aug 04 18:14:39.546700 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.546685    2576 scope.go:117] "RemoveContainer" containerID="4e97b7bb0d580228cb00b40fb77a45bcf0923fba9b4de491fa392bfd9444f984"
Aug 04 18:14:39.546968 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:14:39.546948    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e97b7bb0d580228cb00b40fb77a45bcf0923fba9b4de491fa392bfd9444f984\": container with ID starting with 4e97b7bb0d580228cb00b40fb77a45bcf0923fba9b4de491fa392bfd9444f984 not found: ID does not exist" containerID="4e97b7bb0d580228cb00b40fb77a45bcf0923fba9b4de491fa392bfd9444f984"
Aug 04 18:14:39.547022 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.546967    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e97b7bb0d580228cb00b40fb77a45bcf0923fba9b4de491fa392bfd9444f984"} err="failed to get container status \"4e97b7bb0d580228cb00b40fb77a45bcf0923fba9b4de491fa392bfd9444f984\": rpc error: code = NotFound desc = could not find container \"4e97b7bb0d580228cb00b40fb77a45bcf0923fba9b4de491fa392bfd9444f984\": container with ID starting with 4e97b7bb0d580228cb00b40fb77a45bcf0923fba9b4de491fa392bfd9444f984 not found: ID does not exist"
Aug 04 18:14:39.552663 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.552641    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"]
Aug 04 18:14:39.562581 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.557544    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rp9kk"]
Aug 04 18:14:39.569015 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.568996    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"]
Aug 04 18:14:39.572729 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:39.572710    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewchgk"]
Aug 04 18:14:40.583185 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:40.583153    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b0f28dd-3d13-4ae8-9039-69ed51b761b0" path="/var/lib/kubelet/pods/0b0f28dd-3d13-4ae8-9039-69ed51b761b0/volumes"
Aug 04 18:14:40.583583 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:40.583569    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee39a075-ab6f-4fee-a003-f119676501d1" path="/var/lib/kubelet/pods/ee39a075-ab6f-4fee-a003-f119676501d1/volumes"
Aug 04 18:14:42.349419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:42.349376    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh" podUID="0fa25431-48ac-480e-ac57-2ebb23a715ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:14:52.348892 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:52.348806    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh" podUID="0fa25431-48ac-480e-ac57-2ebb23a715ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:14:53.564736 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:53.564684    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh" podUID="0fa25431-48ac-480e-ac57-2ebb23a715ae" containerName="main" containerID="cri-o://cd95ea171bc37087893dd590d2ddd3951371eda24bc010404769cd6c18dd7527" gracePeriod=45
Aug 04 18:14:53.813839 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:53.813808    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"
Aug 04 18:14:53.872092 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:53.871998    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgf5z\" (UniqueName: \"kubernetes.io/projected/0fa25431-48ac-480e-ac57-2ebb23a715ae-kube-api-access-zgf5z\") pod \"0fa25431-48ac-480e-ac57-2ebb23a715ae\" (UID: \"0fa25431-48ac-480e-ac57-2ebb23a715ae\") "
Aug 04 18:14:53.872092 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:53.872048    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fa25431-48ac-480e-ac57-2ebb23a715ae-tls-certs\") pod \"0fa25431-48ac-480e-ac57-2ebb23a715ae\" (UID: \"0fa25431-48ac-480e-ac57-2ebb23a715ae\") "
Aug 04 18:14:53.874419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:53.874388    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa25431-48ac-480e-ac57-2ebb23a715ae-kube-api-access-zgf5z" (OuterVolumeSpecName: "kube-api-access-zgf5z") pod "0fa25431-48ac-480e-ac57-2ebb23a715ae" (UID: "0fa25431-48ac-480e-ac57-2ebb23a715ae"). InnerVolumeSpecName "kube-api-access-zgf5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:14:53.874582 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:53.874559    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa25431-48ac-480e-ac57-2ebb23a715ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fa25431-48ac-480e-ac57-2ebb23a715ae" (UID: "0fa25431-48ac-480e-ac57-2ebb23a715ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:14:53.973371 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:53.973340    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgf5z\" (UniqueName: \"kubernetes.io/projected/0fa25431-48ac-480e-ac57-2ebb23a715ae-kube-api-access-zgf5z\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:14:53.973371 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:53.973370    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fa25431-48ac-480e-ac57-2ebb23a715ae-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:14:54.601382 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:54.601355    2576 generic.go:356] "Generic (PLEG): container finished" podID="0fa25431-48ac-480e-ac57-2ebb23a715ae" containerID="cd95ea171bc37087893dd590d2ddd3951371eda24bc010404769cd6c18dd7527" exitCode=0
Aug 04 18:14:54.601770 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:54.601399    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"
Aug 04 18:14:54.601770 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:54.601436    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh" event={"ID":"0fa25431-48ac-480e-ac57-2ebb23a715ae","Type":"ContainerDied","Data":"cd95ea171bc37087893dd590d2ddd3951371eda24bc010404769cd6c18dd7527"}
Aug 04 18:14:54.601770 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:54.601475    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh" event={"ID":"0fa25431-48ac-480e-ac57-2ebb23a715ae","Type":"ContainerDied","Data":"95bf077adb679cb4414e777f8c255938531bf8e7c3f083c3552c15836458ad78"}
Aug 04 18:14:54.601770 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:54.601495    2576 scope.go:117] "RemoveContainer" containerID="cd95ea171bc37087893dd590d2ddd3951371eda24bc010404769cd6c18dd7527"
Aug 04 18:14:54.609773 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:54.609757    2576 scope.go:117] "RemoveContainer" containerID="cd95ea171bc37087893dd590d2ddd3951371eda24bc010404769cd6c18dd7527"
Aug 04 18:14:54.610034 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:14:54.610012    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd95ea171bc37087893dd590d2ddd3951371eda24bc010404769cd6c18dd7527\": container with ID starting with cd95ea171bc37087893dd590d2ddd3951371eda24bc010404769cd6c18dd7527 not found: ID does not exist" containerID="cd95ea171bc37087893dd590d2ddd3951371eda24bc010404769cd6c18dd7527"
Aug 04 18:14:54.610099 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:54.610042    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd95ea171bc37087893dd590d2ddd3951371eda24bc010404769cd6c18dd7527"} err="failed to get container status \"cd95ea171bc37087893dd590d2ddd3951371eda24bc010404769cd6c18dd7527\": rpc error: code = NotFound desc = could not find container \"cd95ea171bc37087893dd590d2ddd3951371eda24bc010404769cd6c18dd7527\": container with ID starting with cd95ea171bc37087893dd590d2ddd3951371eda24bc010404769cd6c18dd7527 not found: ID does not exist"
Aug 04 18:14:54.618608 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:54.618586    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"]
Aug 04 18:14:54.621892 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:54.621869    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sr2gh"]
Aug 04 18:14:56.581975 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:56.581937    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fa25431-48ac-480e-ac57-2ebb23a715ae" path="/var/lib/kubelet/pods/0fa25431-48ac-480e-ac57-2ebb23a715ae/volumes"
Aug 04 18:14:58.580906 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:58.580867    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:58.759484 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:58.759451    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:58.847593 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:58.847515    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:58.932182 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:58.932147    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:59.071210 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:59.071173    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:59.165992 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:14:59.165899    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:03.094028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:03.093992    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:05.335303 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:05.335263    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:05.892224 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:05.892182    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:10.657727 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:10.657694    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:11.353896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:11.353856    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:11.453017 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:11.452988    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:12.377969 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.377936    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:12.492056 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.492021    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:12.588701 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.588662    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:12.644280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.644200    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"]
Aug 04 18:15:12.644747 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.644730    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0f28dd-3d13-4ae8-9039-69ed51b761b0" containerName="main"
Aug 04 18:15:12.644792 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.644751    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0f28dd-3d13-4ae8-9039-69ed51b761b0" containerName="main"
Aug 04 18:15:12.644792 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.644772    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee39a075-ab6f-4fee-a003-f119676501d1" containerName="main"
Aug 04 18:15:12.644792 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.644780    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee39a075-ab6f-4fee-a003-f119676501d1" containerName="main"
Aug 04 18:15:12.644896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.644793    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa25431-48ac-480e-ac57-2ebb23a715ae" containerName="main"
Aug 04 18:15:12.644896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.644801    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa25431-48ac-480e-ac57-2ebb23a715ae" containerName="main"
Aug 04 18:15:12.644896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.644875    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0f28dd-3d13-4ae8-9039-69ed51b761b0" containerName="main"
Aug 04 18:15:12.644896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.644887    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee39a075-ab6f-4fee-a003-f119676501d1" containerName="main"
Aug 04 18:15:12.645026 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.644897    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa25431-48ac-480e-ac57-2ebb23a715ae" containerName="main"
Aug 04 18:15:12.649472 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.649453    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.658298 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.658267    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Aug 04 18:15:12.660649 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.660621    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Aug 04 18:15:12.661072 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.661043    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-wvmv7\""
Aug 04 18:15:12.661923 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.661899    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Aug 04 18:15:12.662873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.662853    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"]
Aug 04 18:15:12.707000 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.706970    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:12.749218 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.749181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.750173 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.749263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.750173 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.749310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dxxb\" (UniqueName: \"kubernetes.io/projected/c960d62a-719e-4abc-82f7-8983635a4871-kube-api-access-6dxxb\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.750173 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.749345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c960d62a-719e-4abc-82f7-8983635a4871-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.750173 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.749505    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c960d62a-719e-4abc-82f7-8983635a4871-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.750173 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.749560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.750173 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.749608    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.750173 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.749690    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c960d62a-719e-4abc-82f7-8983635a4871-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.750173 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.749761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.810187 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.810158    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:12.850598 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.850569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.850728 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.850605    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dxxb\" (UniqueName: \"kubernetes.io/projected/c960d62a-719e-4abc-82f7-8983635a4871-kube-api-access-6dxxb\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.850728 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.850624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c960d62a-719e-4abc-82f7-8983635a4871-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.850728 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.850658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c960d62a-719e-4abc-82f7-8983635a4871-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.850728 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.850697    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.850728 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.850723    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.851001 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.850776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c960d62a-719e-4abc-82f7-8983635a4871-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.851001 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.850821    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.851001 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.850846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.851001 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.850994    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.851237 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.851077    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.851237 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.851219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.851317 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.851290    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.851487 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.851466    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c960d62a-719e-4abc-82f7-8983635a4871-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.853037 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.853017    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.853291 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.853272    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c960d62a-719e-4abc-82f7-8983635a4871-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.863237 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.863211    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dxxb\" (UniqueName: \"kubernetes.io/projected/c960d62a-719e-4abc-82f7-8983635a4871-kube-api-access-6dxxb\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.863765 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.863750    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c960d62a-719e-4abc-82f7-8983635a4871-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-pp28g\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:12.969280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:12.969203    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:13.016953 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:13.016922    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:13.108879 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:13.108847    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"]
Aug 04 18:15:13.110919 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:13.110889    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:13.111895 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:15:13.111870    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc960d62a_719e_4abc_82f7_8983635a4871.slice/crio-aa60cbf854e6100775ee3b448fe4db2faff22ecedebd6c72866b9e718e28921a WatchSource:0}: Error finding container aa60cbf854e6100775ee3b448fe4db2faff22ecedebd6c72866b9e718e28921a: Status 404 returned error can't find the container with id aa60cbf854e6100775ee3b448fe4db2faff22ecedebd6c72866b9e718e28921a
Aug 04 18:15:13.694733 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:13.694699    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g" event={"ID":"c960d62a-719e-4abc-82f7-8983635a4871","Type":"ContainerStarted","Data":"aa60cbf854e6100775ee3b448fe4db2faff22ecedebd6c72866b9e718e28921a"}
Aug 04 18:15:15.920838 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:15.920799    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:16.134289 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:16.134239    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:15:16.134439 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:16.134351    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:15:16.134439 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:16.134396    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:15:16.710014 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:16.709977    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g" event={"ID":"c960d62a-719e-4abc-82f7-8983635a4871","Type":"ContainerStarted","Data":"dd65d5eaef3cf34d87ad6c8226cfa20163d5c8a2247ef895dd87460953bfdc24"}
Aug 04 18:15:16.733259 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:16.733035    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g" podStartSLOduration=1.7127534899999999 podStartE2EDuration="4.733017026s" totalImagesPullingTime="3.020263536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:15:16.731857083 +0000 UTC m=+916.707688806" watchObservedRunningTime="2026-08-04 18:15:16.733017026 +0000 UTC m=+916.708848723"
Aug 04 18:15:16.969736 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:16.969645    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:17.407527 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:17.407495    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:17.974513 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:17.974476    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:18.719259 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:18.719224    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:18.720194 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:18.720176    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:22.483271 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:22.483228    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:22.582930 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:22.582889    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:24.163104 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:24.163019    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:24.818350 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:24.818316    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:25.119602 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.119524    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"]
Aug 04 18:15:25.123782 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.123756    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"
Aug 04 18:15:25.127771 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.127744    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-bffjf\""
Aug 04 18:15:25.128106 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.127799    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-lp2pq\""
Aug 04 18:15:25.128242 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.127799    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Aug 04 18:15:25.128242 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.127832    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Aug 04 18:15:25.128242 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.127835    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Aug 04 18:15:25.133581 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.133547    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"]
Aug 04 18:15:25.157869 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.157835    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:25.170128 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.170085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa8b8d0-8d46-4657-897c-ab146aeff20d-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf\" (UID: \"6fa8b8d0-8d46-4657-897c-ab146aeff20d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"
Aug 04 18:15:25.170451 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.170187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qltqf\" (UniqueName: \"kubernetes.io/projected/6fa8b8d0-8d46-4657-897c-ab146aeff20d-kube-api-access-qltqf\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf\" (UID: \"6fa8b8d0-8d46-4657-897c-ab146aeff20d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"
Aug 04 18:15:25.271207 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.271169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qltqf\" (UniqueName: \"kubernetes.io/projected/6fa8b8d0-8d46-4657-897c-ab146aeff20d-kube-api-access-qltqf\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf\" (UID: \"6fa8b8d0-8d46-4657-897c-ab146aeff20d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"
Aug 04 18:15:25.271401 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.271323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa8b8d0-8d46-4657-897c-ab146aeff20d-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf\" (UID: \"6fa8b8d0-8d46-4657-897c-ab146aeff20d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"
Aug 04 18:15:25.273956 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.273926    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa8b8d0-8d46-4657-897c-ab146aeff20d-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf\" (UID: \"6fa8b8d0-8d46-4657-897c-ab146aeff20d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"
Aug 04 18:15:25.281243 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.281221    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qltqf\" (UniqueName: \"kubernetes.io/projected/6fa8b8d0-8d46-4657-897c-ab146aeff20d-kube-api-access-qltqf\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf\" (UID: \"6fa8b8d0-8d46-4657-897c-ab146aeff20d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"
Aug 04 18:15:25.436839 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.436745    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"
Aug 04 18:15:25.577640 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.577617    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"]
Aug 04 18:15:25.755817 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.755731    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf" event={"ID":"6fa8b8d0-8d46-4657-897c-ab146aeff20d","Type":"ContainerStarted","Data":"c6289670978309d9c2817a727d3c283516385e0ea7e7cc8355ca29d629eb0dd7"}
Aug 04 18:15:25.755817 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.755795    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf" event={"ID":"6fa8b8d0-8d46-4657-897c-ab146aeff20d","Type":"ContainerStarted","Data":"94a5b94fddabec093f73532325ca55d16802e5f84de308cc7b28012fc4c607ca"}
Aug 04 18:15:25.756007 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.755827    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"
Aug 04 18:15:25.790415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:25.790367    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf" podStartSLOduration=0.790352815 podStartE2EDuration="790.352815ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:15:25.787180056 +0000 UTC m=+925.763011744" watchObservedRunningTime="2026-08-04 18:15:25.790352815 +0000 UTC m=+925.766184511"
Aug 04 18:15:29.137105 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:29.137065    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:30.047659 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:30.047612    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:30.106574 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:30.106544    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:33.040364 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:33.040300    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:35.525670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:35.525633    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:35.619981 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:35.619948    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:40.521965 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:40.521888    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"]
Aug 04 18:15:40.522373 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:40.522086    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g" podUID="c960d62a-719e-4abc-82f7-8983635a4871" containerName="istio-proxy" containerID="cri-o://dd65d5eaef3cf34d87ad6c8226cfa20163d5c8a2247ef895dd87460953bfdc24" gracePeriod=30
Aug 04 18:15:40.602223 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:40.602180    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:43.371045 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.371014    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:43.434695 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.434661    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-istio-data\") pod \"c960d62a-719e-4abc-82f7-8983635a4871\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") "
Aug 04 18:15:43.434863 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.434779    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c960d62a-719e-4abc-82f7-8983635a4871-istio-podinfo\") pod \"c960d62a-719e-4abc-82f7-8983635a4871\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") "
Aug 04 18:15:43.434934 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.434901    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-workload-certs\") pod \"c960d62a-719e-4abc-82f7-8983635a4871\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") "
Aug 04 18:15:43.434934 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.434911    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-istio-data" (OuterVolumeSpecName: "istio-data") pod "c960d62a-719e-4abc-82f7-8983635a4871" (UID: "c960d62a-719e-4abc-82f7-8983635a4871"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:43.435052 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.434942    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c960d62a-719e-4abc-82f7-8983635a4871-istiod-ca-cert\") pod \"c960d62a-719e-4abc-82f7-8983635a4871\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") "
Aug 04 18:15:43.435052 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.434974    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-workload-socket\") pod \"c960d62a-719e-4abc-82f7-8983635a4871\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") "
Aug 04 18:15:43.435052 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.435026    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c960d62a-719e-4abc-82f7-8983635a4871-istio-token\") pod \"c960d62a-719e-4abc-82f7-8983635a4871\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") "
Aug 04 18:15:43.435367 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.435056    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dxxb\" (UniqueName: \"kubernetes.io/projected/c960d62a-719e-4abc-82f7-8983635a4871-kube-api-access-6dxxb\") pod \"c960d62a-719e-4abc-82f7-8983635a4871\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") "
Aug 04 18:15:43.435367 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.435094    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-istio-envoy\") pod \"c960d62a-719e-4abc-82f7-8983635a4871\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") "
Aug 04 18:15:43.435367 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.435177    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-credential-socket\") pod \"c960d62a-719e-4abc-82f7-8983635a4871\" (UID: \"c960d62a-719e-4abc-82f7-8983635a4871\") "
Aug 04 18:15:43.435367 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.435235    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c960d62a-719e-4abc-82f7-8983635a4871" (UID: "c960d62a-719e-4abc-82f7-8983635a4871"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:43.435367 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.435256    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c960d62a-719e-4abc-82f7-8983635a4871" (UID: "c960d62a-719e-4abc-82f7-8983635a4871"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:43.435815 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.435523    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c960d62a-719e-4abc-82f7-8983635a4871" (UID: "c960d62a-719e-4abc-82f7-8983635a4871"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:43.435815 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.435532    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:15:43.435815 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.435552    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:15:43.435815 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.435566    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:15:43.435815 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.435659    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c960d62a-719e-4abc-82f7-8983635a4871-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c960d62a-719e-4abc-82f7-8983635a4871" (UID: "c960d62a-719e-4abc-82f7-8983635a4871"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:15:43.437629 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.437509    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c960d62a-719e-4abc-82f7-8983635a4871-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c960d62a-719e-4abc-82f7-8983635a4871" (UID: "c960d62a-719e-4abc-82f7-8983635a4871"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 04 18:15:43.437629 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.437587    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c960d62a-719e-4abc-82f7-8983635a4871-istio-token" (OuterVolumeSpecName: "istio-token") pod "c960d62a-719e-4abc-82f7-8983635a4871" (UID: "c960d62a-719e-4abc-82f7-8983635a4871"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:15:43.437789 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.437753    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c960d62a-719e-4abc-82f7-8983635a4871-kube-api-access-6dxxb" (OuterVolumeSpecName: "kube-api-access-6dxxb") pod "c960d62a-719e-4abc-82f7-8983635a4871" (UID: "c960d62a-719e-4abc-82f7-8983635a4871"). InnerVolumeSpecName "kube-api-access-6dxxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:15:43.438190 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.438160    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c960d62a-719e-4abc-82f7-8983635a4871" (UID: "c960d62a-719e-4abc-82f7-8983635a4871"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:43.449276 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.449247    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:43.536758 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.536728    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c960d62a-719e-4abc-82f7-8983635a4871-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:15:43.536758 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.536755    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c960d62a-719e-4abc-82f7-8983635a4871-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:15:43.536959 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.536770    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c960d62a-719e-4abc-82f7-8983635a4871-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:15:43.536959 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.536782    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dxxb\" (UniqueName: \"kubernetes.io/projected/c960d62a-719e-4abc-82f7-8983635a4871-kube-api-access-6dxxb\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:15:43.536959 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.536794    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:15:43.536959 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.536805    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960d62a-719e-4abc-82f7-8983635a4871-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:15:43.845801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.845759    2576 generic.go:356] "Generic (PLEG): container finished" podID="c960d62a-719e-4abc-82f7-8983635a4871" containerID="dd65d5eaef3cf34d87ad6c8226cfa20163d5c8a2247ef895dd87460953bfdc24" exitCode=0
Aug 04 18:15:43.845801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.845802    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"
Aug 04 18:15:43.846001 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.845846    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g" event={"ID":"c960d62a-719e-4abc-82f7-8983635a4871","Type":"ContainerDied","Data":"dd65d5eaef3cf34d87ad6c8226cfa20163d5c8a2247ef895dd87460953bfdc24"}
Aug 04 18:15:43.846001 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.845901    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g" event={"ID":"c960d62a-719e-4abc-82f7-8983635a4871","Type":"ContainerDied","Data":"aa60cbf854e6100775ee3b448fe4db2faff22ecedebd6c72866b9e718e28921a"}
Aug 04 18:15:43.846001 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.845923    2576 scope.go:117] "RemoveContainer" containerID="dd65d5eaef3cf34d87ad6c8226cfa20163d5c8a2247ef895dd87460953bfdc24"
Aug 04 18:15:43.855004 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.854988    2576 scope.go:117] "RemoveContainer" containerID="dd65d5eaef3cf34d87ad6c8226cfa20163d5c8a2247ef895dd87460953bfdc24"
Aug 04 18:15:43.855296 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:15:43.855276    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd65d5eaef3cf34d87ad6c8226cfa20163d5c8a2247ef895dd87460953bfdc24\": container with ID starting with dd65d5eaef3cf34d87ad6c8226cfa20163d5c8a2247ef895dd87460953bfdc24 not found: ID does not exist" containerID="dd65d5eaef3cf34d87ad6c8226cfa20163d5c8a2247ef895dd87460953bfdc24"
Aug 04 18:15:43.855419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.855305    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd65d5eaef3cf34d87ad6c8226cfa20163d5c8a2247ef895dd87460953bfdc24"} err="failed to get container status \"dd65d5eaef3cf34d87ad6c8226cfa20163d5c8a2247ef895dd87460953bfdc24\": rpc error: code = NotFound desc = could not find container \"dd65d5eaef3cf34d87ad6c8226cfa20163d5c8a2247ef895dd87460953bfdc24\": container with ID starting with dd65d5eaef3cf34d87ad6c8226cfa20163d5c8a2247ef895dd87460953bfdc24 not found: ID does not exist"
Aug 04 18:15:43.872102 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.872074    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"]
Aug 04 18:15:43.876002 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:43.875980    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-pp28g"]
Aug 04 18:15:44.582369 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:44.582336    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c960d62a-719e-4abc-82f7-8983635a4871" path="/var/lib/kubelet/pods/c960d62a-719e-4abc-82f7-8983635a4871/volumes"
Aug 04 18:15:45.427000 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:45.426970    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:46.027156 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:46.027092    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:47.384580 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:47.384546    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:52.368624 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:52.368551    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:54.173574 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:54.173537    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:56.761800 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:56.761771    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"
Aug 04 18:15:57.552337 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.552305    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:57.761060 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.761026    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"]
Aug 04 18:15:57.761427 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.761414    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c960d62a-719e-4abc-82f7-8983635a4871" containerName="istio-proxy"
Aug 04 18:15:57.761476 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.761430    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c960d62a-719e-4abc-82f7-8983635a4871" containerName="istio-proxy"
Aug 04 18:15:57.761516 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.761504    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c960d62a-719e-4abc-82f7-8983635a4871" containerName="istio-proxy"
Aug 04 18:15:57.764393 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.764374    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"
Aug 04 18:15:57.767199 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.767177    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Aug 04 18:15:57.768776 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.768756    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Aug 04 18:15:57.768776 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.768766    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-zcs8d\""
Aug 04 18:15:57.768938 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.768786    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Aug 04 18:15:57.768938 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.768817    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-ck4h9\""
Aug 04 18:15:57.772525 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.772505    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"]
Aug 04 18:15:57.789591 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.789566    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:57.864567 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.864502    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89208f8-4155-4f0a-8e3d-5c226eab9189-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c\" (UID: \"c89208f8-4155-4f0a-8e3d-5c226eab9189\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"
Aug 04 18:15:57.864689 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.864635    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2sj5\" (UniqueName: \"kubernetes.io/projected/c89208f8-4155-4f0a-8e3d-5c226eab9189-kube-api-access-d2sj5\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c\" (UID: \"c89208f8-4155-4f0a-8e3d-5c226eab9189\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"
Aug 04 18:15:57.965157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.965097    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2sj5\" (UniqueName: \"kubernetes.io/projected/c89208f8-4155-4f0a-8e3d-5c226eab9189-kube-api-access-d2sj5\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c\" (UID: \"c89208f8-4155-4f0a-8e3d-5c226eab9189\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"
Aug 04 18:15:57.965332 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.965165    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89208f8-4155-4f0a-8e3d-5c226eab9189-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c\" (UID: \"c89208f8-4155-4f0a-8e3d-5c226eab9189\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"
Aug 04 18:15:57.967741 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.967718    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89208f8-4155-4f0a-8e3d-5c226eab9189-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c\" (UID: \"c89208f8-4155-4f0a-8e3d-5c226eab9189\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"
Aug 04 18:15:57.973434 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:57.973410    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2sj5\" (UniqueName: \"kubernetes.io/projected/c89208f8-4155-4f0a-8e3d-5c226eab9189-kube-api-access-d2sj5\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c\" (UID: \"c89208f8-4155-4f0a-8e3d-5c226eab9189\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"
Aug 04 18:15:58.075863 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:58.075828    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"
Aug 04 18:15:58.211957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:58.211931    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"]
Aug 04 18:15:58.214654 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:15:58.214625    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89208f8_4155_4f0a_8e3d_5c226eab9189.slice/crio-d24effac9d7b8c4e57efc5dd05641a7941bc148b0fdc6dbc7ce97ef58aa1c310 WatchSource:0}: Error finding container d24effac9d7b8c4e57efc5dd05641a7941bc148b0fdc6dbc7ce97ef58aa1c310: Status 404 returned error can't find the container with id d24effac9d7b8c4e57efc5dd05641a7941bc148b0fdc6dbc7ce97ef58aa1c310
Aug 04 18:15:58.918847 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:58.918818    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c" event={"ID":"c89208f8-4155-4f0a-8e3d-5c226eab9189","Type":"ContainerStarted","Data":"d5b4dd685e5444b56ea82866ca98e7acc8bd7aa291c47d0c83439bf46ee7e72b"}
Aug 04 18:15:58.919293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:58.918866    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c" event={"ID":"c89208f8-4155-4f0a-8e3d-5c226eab9189","Type":"ContainerStarted","Data":"d24effac9d7b8c4e57efc5dd05641a7941bc148b0fdc6dbc7ce97ef58aa1c310"}
Aug 04 18:15:58.919293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:58.918948    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"
Aug 04 18:15:58.939190 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:15:58.939128    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c" podStartSLOduration=1.93907017 podStartE2EDuration="1.93907017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:15:58.937165406 +0000 UTC m=+958.912997103" watchObservedRunningTime="2026-08-04 18:15:58.93907017 +0000 UTC m=+958.914901870"
Aug 04 18:16:11.432328 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:11.432296    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:12.983566 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:12.983529    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:13.007714 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:13.007682    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"]
Aug 04 18:16:13.047880 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:13.047840    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:16.762322 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:16.762268    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf" podUID="6fa8b8d0-8d46-4657-897c-ab146aeff20d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:16:26.762132 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:26.762024    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf" podUID="6fa8b8d0-8d46-4657-897c-ab146aeff20d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:16:28.027729 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:28.027687    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf" podUID="6fa8b8d0-8d46-4657-897c-ab146aeff20d" containerName="main" containerID="cri-o://c6289670978309d9c2817a727d3c283516385e0ea7e7cc8355ca29d629eb0dd7" gracePeriod=45
Aug 04 18:16:28.282853 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:28.282790    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"
Aug 04 18:16:28.319706 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:28.319676    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa8b8d0-8d46-4657-897c-ab146aeff20d-tls-certs\") pod \"6fa8b8d0-8d46-4657-897c-ab146aeff20d\" (UID: \"6fa8b8d0-8d46-4657-897c-ab146aeff20d\") "
Aug 04 18:16:28.319828 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:28.319723    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qltqf\" (UniqueName: \"kubernetes.io/projected/6fa8b8d0-8d46-4657-897c-ab146aeff20d-kube-api-access-qltqf\") pod \"6fa8b8d0-8d46-4657-897c-ab146aeff20d\" (UID: \"6fa8b8d0-8d46-4657-897c-ab146aeff20d\") "
Aug 04 18:16:28.321834 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:28.321796    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa8b8d0-8d46-4657-897c-ab146aeff20d-kube-api-access-qltqf" (OuterVolumeSpecName: "kube-api-access-qltqf") pod "6fa8b8d0-8d46-4657-897c-ab146aeff20d" (UID: "6fa8b8d0-8d46-4657-897c-ab146aeff20d"). InnerVolumeSpecName "kube-api-access-qltqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:16:28.321834 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:28.321802    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa8b8d0-8d46-4657-897c-ab146aeff20d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fa8b8d0-8d46-4657-897c-ab146aeff20d" (UID: "6fa8b8d0-8d46-4657-897c-ab146aeff20d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:16:28.420989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:28.420948    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qltqf\" (UniqueName: \"kubernetes.io/projected/6fa8b8d0-8d46-4657-897c-ab146aeff20d-kube-api-access-qltqf\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:16:28.420989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:28.420984    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa8b8d0-8d46-4657-897c-ab146aeff20d-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:16:29.067584 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:29.067551    2576 generic.go:356] "Generic (PLEG): container finished" podID="6fa8b8d0-8d46-4657-897c-ab146aeff20d" containerID="c6289670978309d9c2817a727d3c283516385e0ea7e7cc8355ca29d629eb0dd7" exitCode=0
Aug 04 18:16:29.067978 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:29.067590    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"
Aug 04 18:16:29.067978 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:29.067635    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf" event={"ID":"6fa8b8d0-8d46-4657-897c-ab146aeff20d","Type":"ContainerDied","Data":"c6289670978309d9c2817a727d3c283516385e0ea7e7cc8355ca29d629eb0dd7"}
Aug 04 18:16:29.067978 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:29.067685    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf" event={"ID":"6fa8b8d0-8d46-4657-897c-ab146aeff20d","Type":"ContainerDied","Data":"94a5b94fddabec093f73532325ca55d16802e5f84de308cc7b28012fc4c607ca"}
Aug 04 18:16:29.067978 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:29.067710    2576 scope.go:117] "RemoveContainer" containerID="c6289670978309d9c2817a727d3c283516385e0ea7e7cc8355ca29d629eb0dd7"
Aug 04 18:16:29.075973 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:29.075953    2576 scope.go:117] "RemoveContainer" containerID="c6289670978309d9c2817a727d3c283516385e0ea7e7cc8355ca29d629eb0dd7"
Aug 04 18:16:29.076222 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:16:29.076202    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6289670978309d9c2817a727d3c283516385e0ea7e7cc8355ca29d629eb0dd7\": container with ID starting with c6289670978309d9c2817a727d3c283516385e0ea7e7cc8355ca29d629eb0dd7 not found: ID does not exist" containerID="c6289670978309d9c2817a727d3c283516385e0ea7e7cc8355ca29d629eb0dd7"
Aug 04 18:16:29.076272 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:29.076230    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6289670978309d9c2817a727d3c283516385e0ea7e7cc8355ca29d629eb0dd7"} err="failed to get container status \"c6289670978309d9c2817a727d3c283516385e0ea7e7cc8355ca29d629eb0dd7\": rpc error: code = NotFound desc = could not find container \"c6289670978309d9c2817a727d3c283516385e0ea7e7cc8355ca29d629eb0dd7\": container with ID starting with c6289670978309d9c2817a727d3c283516385e0ea7e7cc8355ca29d629eb0dd7 not found: ID does not exist"
Aug 04 18:16:29.091404 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:29.091374    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"]
Aug 04 18:16:29.096085 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:29.096059    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5xqjsf"]
Aug 04 18:16:29.926087 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:29.926059    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"
Aug 04 18:16:30.582195 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:30.582163    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fa8b8d0-8d46-4657-897c-ab146aeff20d" path="/var/lib/kubelet/pods/6fa8b8d0-8d46-4657-897c-ab146aeff20d/volumes"
Aug 04 18:16:31.269098 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:31.269066    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:31.366148 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:31.366097    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:37.417947 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:37.417911    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:42.996527 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:42.996490    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:43.063938 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.063893    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:43.316786 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.316746    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:43.549732 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.548176    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"]
Aug 04 18:16:43.549732 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.549046    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fa8b8d0-8d46-4657-897c-ab146aeff20d" containerName="main"
Aug 04 18:16:43.549732 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.549067    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fa8b8d0-8d46-4657-897c-ab146aeff20d" containerName="main"
Aug 04 18:16:43.551036 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.550212    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fa8b8d0-8d46-4657-897c-ab146aeff20d" containerName="main"
Aug 04 18:16:43.555325 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.555297    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"
Aug 04 18:16:43.559257 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.559232    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-xflt5\""
Aug 04 18:16:43.559549 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.559296    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Aug 04 18:16:43.559549 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.559407    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"]
Aug 04 18:16:43.559723 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.559423    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-j9vvv\""
Aug 04 18:16:43.559723 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.559464    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Aug 04 18:16:43.559723 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.559481    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Aug 04 18:16:43.582419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.582353    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:43.654733 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.654700    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6af713f8-d796-4a99-9f02-ce0e7bb58590-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm\" (UID: \"6af713f8-d796-4a99-9f02-ce0e7bb58590\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"
Aug 04 18:16:43.654884 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.654743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clxh4\" (UniqueName: \"kubernetes.io/projected/6af713f8-d796-4a99-9f02-ce0e7bb58590-kube-api-access-clxh4\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm\" (UID: \"6af713f8-d796-4a99-9f02-ce0e7bb58590\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"
Aug 04 18:16:43.755787 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.755756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6af713f8-d796-4a99-9f02-ce0e7bb58590-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm\" (UID: \"6af713f8-d796-4a99-9f02-ce0e7bb58590\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"
Aug 04 18:16:43.755963 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.755798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clxh4\" (UniqueName: \"kubernetes.io/projected/6af713f8-d796-4a99-9f02-ce0e7bb58590-kube-api-access-clxh4\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm\" (UID: \"6af713f8-d796-4a99-9f02-ce0e7bb58590\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"
Aug 04 18:16:43.758238 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.758210    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6af713f8-d796-4a99-9f02-ce0e7bb58590-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm\" (UID: \"6af713f8-d796-4a99-9f02-ce0e7bb58590\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"
Aug 04 18:16:43.764626 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.764601    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clxh4\" (UniqueName: \"kubernetes.io/projected/6af713f8-d796-4a99-9f02-ce0e7bb58590-kube-api-access-clxh4\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm\" (UID: \"6af713f8-d796-4a99-9f02-ce0e7bb58590\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"
Aug 04 18:16:43.869040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:43.868951    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"
Aug 04 18:16:44.004296 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:44.004183    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"]
Aug 04 18:16:44.006760 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:16:44.006728    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af713f8_d796_4a99_9f02_ce0e7bb58590.slice/crio-3f275a3476113dfbb5c3905fa022a871fa5fb516ce1ddfcd6a9ee8f238b0371e WatchSource:0}: Error finding container 3f275a3476113dfbb5c3905fa022a871fa5fb516ce1ddfcd6a9ee8f238b0371e: Status 404 returned error can't find the container with id 3f275a3476113dfbb5c3905fa022a871fa5fb516ce1ddfcd6a9ee8f238b0371e
Aug 04 18:16:44.147813 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:44.147729    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm" event={"ID":"6af713f8-d796-4a99-9f02-ce0e7bb58590","Type":"ContainerStarted","Data":"98ba2e26e43b5bf2f0d69ab42827ff3bf619d9baa8d47e4a03de091796179e7b"}
Aug 04 18:16:44.147813 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:44.147781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm" event={"ID":"6af713f8-d796-4a99-9f02-ce0e7bb58590","Type":"ContainerStarted","Data":"3f275a3476113dfbb5c3905fa022a871fa5fb516ce1ddfcd6a9ee8f238b0371e"}
Aug 04 18:16:44.148012 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:44.147831    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"
Aug 04 18:16:44.165609 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:44.165555    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm" podStartSLOduration=1.165538556 podStartE2EDuration="1.165538556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:16:44.1638647 +0000 UTC m=+1004.139696412" watchObservedRunningTime="2026-08-04 18:16:44.165538556 +0000 UTC m=+1004.141370253"
Aug 04 18:16:51.916397 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:16:51.916363    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:15.153780 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:15.153750    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"
Aug 04 18:17:16.856273 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:16.856235    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"]
Aug 04 18:17:16.863710 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:16.863679    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:16.924414 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:16.924377    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:25.154082 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:25.154031    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm" podUID="6af713f8-d796-4a99-9f02-ce0e7bb58590" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:17:31.875960 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:31.875912    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm" podUID="6af713f8-d796-4a99-9f02-ce0e7bb58590" containerName="main" containerID="cri-o://98ba2e26e43b5bf2f0d69ab42827ff3bf619d9baa8d47e4a03de091796179e7b" gracePeriod=45
Aug 04 18:17:32.121194 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.121166    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"
Aug 04 18:17:32.193535 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.193449    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6af713f8-d796-4a99-9f02-ce0e7bb58590-tls-certs\") pod \"6af713f8-d796-4a99-9f02-ce0e7bb58590\" (UID: \"6af713f8-d796-4a99-9f02-ce0e7bb58590\") "
Aug 04 18:17:32.193685 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.193561    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clxh4\" (UniqueName: \"kubernetes.io/projected/6af713f8-d796-4a99-9f02-ce0e7bb58590-kube-api-access-clxh4\") pod \"6af713f8-d796-4a99-9f02-ce0e7bb58590\" (UID: \"6af713f8-d796-4a99-9f02-ce0e7bb58590\") "
Aug 04 18:17:32.195696 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.195671    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af713f8-d796-4a99-9f02-ce0e7bb58590-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6af713f8-d796-4a99-9f02-ce0e7bb58590" (UID: "6af713f8-d796-4a99-9f02-ce0e7bb58590"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:17:32.195758 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.195670    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af713f8-d796-4a99-9f02-ce0e7bb58590-kube-api-access-clxh4" (OuterVolumeSpecName: "kube-api-access-clxh4") pod "6af713f8-d796-4a99-9f02-ce0e7bb58590" (UID: "6af713f8-d796-4a99-9f02-ce0e7bb58590"). InnerVolumeSpecName "kube-api-access-clxh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:17:32.294967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.294927    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clxh4\" (UniqueName: \"kubernetes.io/projected/6af713f8-d796-4a99-9f02-ce0e7bb58590-kube-api-access-clxh4\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:17:32.294967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.294959    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6af713f8-d796-4a99-9f02-ce0e7bb58590-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:17:32.375207 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.375166    2576 generic.go:356] "Generic (PLEG): container finished" podID="6af713f8-d796-4a99-9f02-ce0e7bb58590" containerID="98ba2e26e43b5bf2f0d69ab42827ff3bf619d9baa8d47e4a03de091796179e7b" exitCode=0
Aug 04 18:17:32.375372 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.375229    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"
Aug 04 18:17:32.375372 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.375252    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm" event={"ID":"6af713f8-d796-4a99-9f02-ce0e7bb58590","Type":"ContainerDied","Data":"98ba2e26e43b5bf2f0d69ab42827ff3bf619d9baa8d47e4a03de091796179e7b"}
Aug 04 18:17:32.375372 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.375330    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm" event={"ID":"6af713f8-d796-4a99-9f02-ce0e7bb58590","Type":"ContainerDied","Data":"3f275a3476113dfbb5c3905fa022a871fa5fb516ce1ddfcd6a9ee8f238b0371e"}
Aug 04 18:17:32.375372 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.375354    2576 scope.go:117] "RemoveContainer" containerID="98ba2e26e43b5bf2f0d69ab42827ff3bf619d9baa8d47e4a03de091796179e7b"
Aug 04 18:17:32.386347 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.386322    2576 scope.go:117] "RemoveContainer" containerID="98ba2e26e43b5bf2f0d69ab42827ff3bf619d9baa8d47e4a03de091796179e7b"
Aug 04 18:17:32.386652 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:17:32.386623    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ba2e26e43b5bf2f0d69ab42827ff3bf619d9baa8d47e4a03de091796179e7b\": container with ID starting with 98ba2e26e43b5bf2f0d69ab42827ff3bf619d9baa8d47e4a03de091796179e7b not found: ID does not exist" containerID="98ba2e26e43b5bf2f0d69ab42827ff3bf619d9baa8d47e4a03de091796179e7b"
Aug 04 18:17:32.386710 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.386667    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ba2e26e43b5bf2f0d69ab42827ff3bf619d9baa8d47e4a03de091796179e7b"} err="failed to get container status \"98ba2e26e43b5bf2f0d69ab42827ff3bf619d9baa8d47e4a03de091796179e7b\": rpc error: code = NotFound desc = could not find container \"98ba2e26e43b5bf2f0d69ab42827ff3bf619d9baa8d47e4a03de091796179e7b\": container with ID starting with 98ba2e26e43b5bf2f0d69ab42827ff3bf619d9baa8d47e4a03de091796179e7b not found: ID does not exist"
Aug 04 18:17:32.401707 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.401676    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"]
Aug 04 18:17:32.405666 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.405640    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekcfmm"]
Aug 04 18:17:32.583019 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:32.582987    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6af713f8-d796-4a99-9f02-ce0e7bb58590" path="/var/lib/kubelet/pods/6af713f8-d796-4a99-9f02-ce0e7bb58590/volumes"
Aug 04 18:17:36.857867 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:36.857810    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:36.955543 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:36.955506    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:43.147982 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.147940    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:43.436158 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.436055    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:43.616333 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.616298    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"]
Aug 04 18:17:43.616687 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.616672    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6af713f8-d796-4a99-9f02-ce0e7bb58590" containerName="main"
Aug 04 18:17:43.616687 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.616687    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6af713f8-d796-4a99-9f02-ce0e7bb58590" containerName="main"
Aug 04 18:17:43.616843 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.616762    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6af713f8-d796-4a99-9f02-ce0e7bb58590" containerName="main"
Aug 04 18:17:43.619548 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.619529    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"
Aug 04 18:17:43.623376 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.623357    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-nlj6g\""
Aug 04 18:17:43.623492 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.623382    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Aug 04 18:17:43.623492 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.623391    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Aug 04 18:17:43.623492 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.623438    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-4zv9l\""
Aug 04 18:17:43.623684 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.623529    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Aug 04 18:17:43.628859 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.628836    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"]
Aug 04 18:17:43.640998 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.640967    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:43.701158 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.701061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c92d194-13b7-46c6-ba53-9b10a15b2b67-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs\" (UID: \"8c92d194-13b7-46c6-ba53-9b10a15b2b67\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"
Aug 04 18:17:43.701300 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.701187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz4lv\" (UniqueName: \"kubernetes.io/projected/8c92d194-13b7-46c6-ba53-9b10a15b2b67-kube-api-access-vz4lv\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs\" (UID: \"8c92d194-13b7-46c6-ba53-9b10a15b2b67\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"
Aug 04 18:17:43.802470 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.802434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz4lv\" (UniqueName: \"kubernetes.io/projected/8c92d194-13b7-46c6-ba53-9b10a15b2b67-kube-api-access-vz4lv\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs\" (UID: \"8c92d194-13b7-46c6-ba53-9b10a15b2b67\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"
Aug 04 18:17:43.802642 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.802503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c92d194-13b7-46c6-ba53-9b10a15b2b67-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs\" (UID: \"8c92d194-13b7-46c6-ba53-9b10a15b2b67\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"
Aug 04 18:17:43.804879 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.804856    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c92d194-13b7-46c6-ba53-9b10a15b2b67-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs\" (UID: \"8c92d194-13b7-46c6-ba53-9b10a15b2b67\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"
Aug 04 18:17:43.812275 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.812251    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz4lv\" (UniqueName: \"kubernetes.io/projected/8c92d194-13b7-46c6-ba53-9b10a15b2b67-kube-api-access-vz4lv\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs\" (UID: \"8c92d194-13b7-46c6-ba53-9b10a15b2b67\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"
Aug 04 18:17:43.930741 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:43.930707    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"
Aug 04 18:17:44.072929 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:44.072891    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"]
Aug 04 18:17:44.074076 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:17:44.074050    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c92d194_13b7_46c6_ba53_9b10a15b2b67.slice/crio-823baf1ac5594f52d56eb41f6776616b0f00f1d6678cfc8b90a62d0b916ddbde WatchSource:0}: Error finding container 823baf1ac5594f52d56eb41f6776616b0f00f1d6678cfc8b90a62d0b916ddbde: Status 404 returned error can't find the container with id 823baf1ac5594f52d56eb41f6776616b0f00f1d6678cfc8b90a62d0b916ddbde
Aug 04 18:17:44.432406 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:44.432375    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs" event={"ID":"8c92d194-13b7-46c6-ba53-9b10a15b2b67","Type":"ContainerStarted","Data":"56d26ce4b7b6324b2a9410dd7b6cb6228c8b19c8d544d0bb744f16c4e93b16eb"}
Aug 04 18:17:44.432807 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:44.432452    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs" event={"ID":"8c92d194-13b7-46c6-ba53-9b10a15b2b67","Type":"ContainerStarted","Data":"823baf1ac5594f52d56eb41f6776616b0f00f1d6678cfc8b90a62d0b916ddbde"}
Aug 04 18:17:44.432807 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:44.432483    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"
Aug 04 18:17:44.452669 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:44.452601    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs" podStartSLOduration=1.452583663 podStartE2EDuration="1.452583663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:44.450672472 +0000 UTC m=+1064.426504172" watchObservedRunningTime="2026-08-04 18:17:44.452583663 +0000 UTC m=+1064.428415359"
Aug 04 18:17:49.310383 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:49.310346    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:49.380567 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:49.380525    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:53.044529 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:53.044420    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:53.764776 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:17:53.764740    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:02.992354 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:02.992258    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:02.998163 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:02.998131    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"]
Aug 04 18:18:03.091004 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:03.090965    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:09.925891 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:09.925853    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c" podUID="c89208f8-4155-4f0a-8e3d-5c226eab9189" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:18:15.438452 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:15.438420    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"
Aug 04 18:18:17.002180 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:17.002143    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"]
Aug 04 18:18:17.013101 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:17.013061    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:17.110087 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:17.110055    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:18.017541 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.017497    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c" podUID="c89208f8-4155-4f0a-8e3d-5c226eab9189" containerName="main" containerID="cri-o://d5b4dd685e5444b56ea82866ca98e7acc8bd7aa291c47d0c83439bf46ee7e72b" gracePeriod=45
Aug 04 18:18:18.267852 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.267784    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"
Aug 04 18:18:18.419655 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.419612    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2sj5\" (UniqueName: \"kubernetes.io/projected/c89208f8-4155-4f0a-8e3d-5c226eab9189-kube-api-access-d2sj5\") pod \"c89208f8-4155-4f0a-8e3d-5c226eab9189\" (UID: \"c89208f8-4155-4f0a-8e3d-5c226eab9189\") "
Aug 04 18:18:18.419854 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.419703    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89208f8-4155-4f0a-8e3d-5c226eab9189-tls-certs\") pod \"c89208f8-4155-4f0a-8e3d-5c226eab9189\" (UID: \"c89208f8-4155-4f0a-8e3d-5c226eab9189\") "
Aug 04 18:18:18.421806 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.421777    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89208f8-4155-4f0a-8e3d-5c226eab9189-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c89208f8-4155-4f0a-8e3d-5c226eab9189" (UID: "c89208f8-4155-4f0a-8e3d-5c226eab9189"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:18:18.421806 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.421778    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89208f8-4155-4f0a-8e3d-5c226eab9189-kube-api-access-d2sj5" (OuterVolumeSpecName: "kube-api-access-d2sj5") pod "c89208f8-4155-4f0a-8e3d-5c226eab9189" (UID: "c89208f8-4155-4f0a-8e3d-5c226eab9189"). InnerVolumeSpecName "kube-api-access-d2sj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:18:18.520832 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.520751    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2sj5\" (UniqueName: \"kubernetes.io/projected/c89208f8-4155-4f0a-8e3d-5c226eab9189-kube-api-access-d2sj5\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:18:18.520832 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.520777    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89208f8-4155-4f0a-8e3d-5c226eab9189-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:18:18.596030 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.595985    2576 generic.go:356] "Generic (PLEG): container finished" podID="c89208f8-4155-4f0a-8e3d-5c226eab9189" containerID="d5b4dd685e5444b56ea82866ca98e7acc8bd7aa291c47d0c83439bf46ee7e72b" exitCode=0
Aug 04 18:18:18.596198 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.596081    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"
Aug 04 18:18:18.596198 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.596080    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c" event={"ID":"c89208f8-4155-4f0a-8e3d-5c226eab9189","Type":"ContainerDied","Data":"d5b4dd685e5444b56ea82866ca98e7acc8bd7aa291c47d0c83439bf46ee7e72b"}
Aug 04 18:18:18.596323 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.596241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c" event={"ID":"c89208f8-4155-4f0a-8e3d-5c226eab9189","Type":"ContainerDied","Data":"d24effac9d7b8c4e57efc5dd05641a7941bc148b0fdc6dbc7ce97ef58aa1c310"}
Aug 04 18:18:18.596323 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.596281    2576 scope.go:117] "RemoveContainer" containerID="d5b4dd685e5444b56ea82866ca98e7acc8bd7aa291c47d0c83439bf46ee7e72b"
Aug 04 18:18:18.604812 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.604790    2576 scope.go:117] "RemoveContainer" containerID="d5b4dd685e5444b56ea82866ca98e7acc8bd7aa291c47d0c83439bf46ee7e72b"
Aug 04 18:18:18.605057 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:18:18.605035    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b4dd685e5444b56ea82866ca98e7acc8bd7aa291c47d0c83439bf46ee7e72b\": container with ID starting with d5b4dd685e5444b56ea82866ca98e7acc8bd7aa291c47d0c83439bf46ee7e72b not found: ID does not exist" containerID="d5b4dd685e5444b56ea82866ca98e7acc8bd7aa291c47d0c83439bf46ee7e72b"
Aug 04 18:18:18.605128 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.605064    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b4dd685e5444b56ea82866ca98e7acc8bd7aa291c47d0c83439bf46ee7e72b"} err="failed to get container status \"d5b4dd685e5444b56ea82866ca98e7acc8bd7aa291c47d0c83439bf46ee7e72b\": rpc error: code = NotFound desc = could not find container \"d5b4dd685e5444b56ea82866ca98e7acc8bd7aa291c47d0c83439bf46ee7e72b\": container with ID starting with d5b4dd685e5444b56ea82866ca98e7acc8bd7aa291c47d0c83439bf46ee7e72b not found: ID does not exist"
Aug 04 18:18:18.618128 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.618089    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"]
Aug 04 18:18:18.623916 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:18.623882    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-q968c"]
Aug 04 18:18:20.581598 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:20.581566    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c89208f8-4155-4f0a-8e3d-5c226eab9189" path="/var/lib/kubelet/pods/c89208f8-4155-4f0a-8e3d-5c226eab9189/volumes"
Aug 04 18:18:21.680582 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:21.680545    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:23.066506 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:23.066453    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:23.184945 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:23.184915    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:24.031713 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:24.031680    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:25.438804 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:25.438763    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs" podUID="8c92d194-13b7-46c6-ba53-9b10a15b2b67" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:18:27.015731 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:27.015693    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:30.275453 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:30.275415    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:32.022878 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.022840    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs" podUID="8c92d194-13b7-46c6-ba53-9b10a15b2b67" containerName="main" containerID="cri-o://56d26ce4b7b6324b2a9410dd7b6cb6228c8b19c8d544d0bb744f16c4e93b16eb" gracePeriod=45
Aug 04 18:18:32.274587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.274524    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"
Aug 04 18:18:32.452961 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.452920    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c92d194-13b7-46c6-ba53-9b10a15b2b67-tls-certs\") pod \"8c92d194-13b7-46c6-ba53-9b10a15b2b67\" (UID: \"8c92d194-13b7-46c6-ba53-9b10a15b2b67\") "
Aug 04 18:18:32.453172 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.453011    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz4lv\" (UniqueName: \"kubernetes.io/projected/8c92d194-13b7-46c6-ba53-9b10a15b2b67-kube-api-access-vz4lv\") pod \"8c92d194-13b7-46c6-ba53-9b10a15b2b67\" (UID: \"8c92d194-13b7-46c6-ba53-9b10a15b2b67\") "
Aug 04 18:18:32.455310 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.455276    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c92d194-13b7-46c6-ba53-9b10a15b2b67-kube-api-access-vz4lv" (OuterVolumeSpecName: "kube-api-access-vz4lv") pod "8c92d194-13b7-46c6-ba53-9b10a15b2b67" (UID: "8c92d194-13b7-46c6-ba53-9b10a15b2b67"). InnerVolumeSpecName "kube-api-access-vz4lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:18:32.455310 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.455278    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c92d194-13b7-46c6-ba53-9b10a15b2b67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c92d194-13b7-46c6-ba53-9b10a15b2b67" (UID: "8c92d194-13b7-46c6-ba53-9b10a15b2b67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:18:32.554491 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.554455    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c92d194-13b7-46c6-ba53-9b10a15b2b67-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:18:32.554491 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.554485    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz4lv\" (UniqueName: \"kubernetes.io/projected/8c92d194-13b7-46c6-ba53-9b10a15b2b67-kube-api-access-vz4lv\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:18:32.662832 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.662796    2576 generic.go:356] "Generic (PLEG): container finished" podID="8c92d194-13b7-46c6-ba53-9b10a15b2b67" containerID="56d26ce4b7b6324b2a9410dd7b6cb6228c8b19c8d544d0bb744f16c4e93b16eb" exitCode=0
Aug 04 18:18:32.663027 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.662842    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"
Aug 04 18:18:32.663027 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.662885    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs" event={"ID":"8c92d194-13b7-46c6-ba53-9b10a15b2b67","Type":"ContainerDied","Data":"56d26ce4b7b6324b2a9410dd7b6cb6228c8b19c8d544d0bb744f16c4e93b16eb"}
Aug 04 18:18:32.663027 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.662943    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs" event={"ID":"8c92d194-13b7-46c6-ba53-9b10a15b2b67","Type":"ContainerDied","Data":"823baf1ac5594f52d56eb41f6776616b0f00f1d6678cfc8b90a62d0b916ddbde"}
Aug 04 18:18:32.663027 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.662970    2576 scope.go:117] "RemoveContainer" containerID="56d26ce4b7b6324b2a9410dd7b6cb6228c8b19c8d544d0bb744f16c4e93b16eb"
Aug 04 18:18:32.671404 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.671386    2576 scope.go:117] "RemoveContainer" containerID="56d26ce4b7b6324b2a9410dd7b6cb6228c8b19c8d544d0bb744f16c4e93b16eb"
Aug 04 18:18:32.671665 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:18:32.671643    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d26ce4b7b6324b2a9410dd7b6cb6228c8b19c8d544d0bb744f16c4e93b16eb\": container with ID starting with 56d26ce4b7b6324b2a9410dd7b6cb6228c8b19c8d544d0bb744f16c4e93b16eb not found: ID does not exist" containerID="56d26ce4b7b6324b2a9410dd7b6cb6228c8b19c8d544d0bb744f16c4e93b16eb"
Aug 04 18:18:32.671712 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.671674    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d26ce4b7b6324b2a9410dd7b6cb6228c8b19c8d544d0bb744f16c4e93b16eb"} err="failed to get container status \"56d26ce4b7b6324b2a9410dd7b6cb6228c8b19c8d544d0bb744f16c4e93b16eb\": rpc error: code = NotFound desc = could not find container \"56d26ce4b7b6324b2a9410dd7b6cb6228c8b19c8d544d0bb744f16c4e93b16eb\": container with ID starting with 56d26ce4b7b6324b2a9410dd7b6cb6228c8b19c8d544d0bb744f16c4e93b16eb not found: ID does not exist"
Aug 04 18:18:32.685215 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.685185    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"]
Aug 04 18:18:32.688858 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:32.688832    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565878phxs"]
Aug 04 18:18:34.582009 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:34.581972    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c92d194-13b7-46c6-ba53-9b10a15b2b67" path="/var/lib/kubelet/pods/8c92d194-13b7-46c6-ba53-9b10a15b2b67/volumes"
Aug 04 18:18:35.795333 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:35.795300    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:35.987885 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:35.987851    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"]
Aug 04 18:18:35.988263 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:35.988247    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c92d194-13b7-46c6-ba53-9b10a15b2b67" containerName="main"
Aug 04 18:18:35.988312 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:35.988268    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c92d194-13b7-46c6-ba53-9b10a15b2b67" containerName="main"
Aug 04 18:18:35.988312 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:35.988284    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c89208f8-4155-4f0a-8e3d-5c226eab9189" containerName="main"
Aug 04 18:18:35.988312 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:35.988290    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c89208f8-4155-4f0a-8e3d-5c226eab9189" containerName="main"
Aug 04 18:18:35.988408 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:35.988355    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c92d194-13b7-46c6-ba53-9b10a15b2b67" containerName="main"
Aug 04 18:18:35.988408 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:35.988367    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c89208f8-4155-4f0a-8e3d-5c226eab9189" containerName="main"
Aug 04 18:18:35.992688 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:35.992666    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"
Aug 04 18:18:35.998637 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:35.998607    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Aug 04 18:18:35.998785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:35.998652    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Aug 04 18:18:35.998785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:35.998671    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Aug 04 18:18:35.998785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:35.998674    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-c9q9g\""
Aug 04 18:18:35.998785 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:35.998607    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-zhhwt\""
Aug 04 18:18:36.015815 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.015783    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"]
Aug 04 18:18:36.048550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.048466    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:36.084784 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.084744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpc4c\" (UniqueName: \"kubernetes.io/projected/c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3-kube-api-access-hpc4c\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx\" (UID: \"c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"
Aug 04 18:18:36.084946 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.084862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx\" (UID: \"c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"
Aug 04 18:18:36.186094 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.186057    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx\" (UID: \"c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"
Aug 04 18:18:36.186321 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.186148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpc4c\" (UniqueName: \"kubernetes.io/projected/c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3-kube-api-access-hpc4c\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx\" (UID: \"c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"
Aug 04 18:18:36.188593 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.188567    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx\" (UID: \"c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"
Aug 04 18:18:36.196585 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.196564    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpc4c\" (UniqueName: \"kubernetes.io/projected/c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3-kube-api-access-hpc4c\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx\" (UID: \"c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"
Aug 04 18:18:36.303476 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.303395    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"
Aug 04 18:18:36.442025 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.441990    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"]
Aug 04 18:18:36.446444 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:18:36.446419    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83b83f7_a1ae_4806_b5cf_fd9e1c89d5b3.slice/crio-25c3e77591d40620b60b68c3584e49d468f74b9eb1f1f55c9e3b122dbb1308d6 WatchSource:0}: Error finding container 25c3e77591d40620b60b68c3584e49d468f74b9eb1f1f55c9e3b122dbb1308d6: Status 404 returned error can't find the container with id 25c3e77591d40620b60b68c3584e49d468f74b9eb1f1f55c9e3b122dbb1308d6
Aug 04 18:18:36.688229 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.688141    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx" event={"ID":"c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3","Type":"ContainerStarted","Data":"8467053ef044f66cae54eef6d15fc160baa4476181f23af418562f3d20b9e30b"}
Aug 04 18:18:36.688229 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.688200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx" event={"ID":"c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3","Type":"ContainerStarted","Data":"25c3e77591d40620b60b68c3584e49d468f74b9eb1f1f55c9e3b122dbb1308d6"}
Aug 04 18:18:36.688412 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.688263    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"
Aug 04 18:18:36.709053 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.708976    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx" podStartSLOduration=1.708955838 podStartE2EDuration="1.708955838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:18:36.707613534 +0000 UTC m=+1116.683445233" watchObservedRunningTime="2026-08-04 18:18:36.708955838 +0000 UTC m=+1116.684787536"
Aug 04 18:18:36.983450 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:36.983367    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:37.080661 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:37.080623    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:37.494100 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:37.494066    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:37.561016 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:37.560980    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:39.589796 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:39.589765    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:40.518027 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:40.517992    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:40.618578 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:40.618545    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:43.118693 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:43.118654    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:48.649524 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:48.649490    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:48.714065 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:48.714031    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:50.045611 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:18:50.045576    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:00.719470 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:00.719434    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:00.944277 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:00.944182    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"]
Aug 04 18:19:00.948986 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:00.948960    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"
Aug 04 18:19:00.951774 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:00.951746    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Aug 04 18:19:00.952792 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:00.952770    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-ztqzn\""
Aug 04 18:19:00.952929 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:00.952768    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Aug 04 18:19:00.952929 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:00.952811    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Aug 04 18:19:00.952929 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:00.952813    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-lhf68\""
Aug 04 18:19:00.958581 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:00.958558    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"]
Aug 04 18:19:00.985529 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:00.985494    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:01.102028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:01.101974    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ec0ad61-2e29-4aeb-8f2b-00db33de0861-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s\" (UID: \"8ec0ad61-2e29-4aeb-8f2b-00db33de0861\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"
Aug 04 18:19:01.102242 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:01.102158    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h87w\" (UniqueName: \"kubernetes.io/projected/8ec0ad61-2e29-4aeb-8f2b-00db33de0861-kube-api-access-7h87w\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s\" (UID: \"8ec0ad61-2e29-4aeb-8f2b-00db33de0861\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"
Aug 04 18:19:01.203149 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:01.203034    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h87w\" (UniqueName: \"kubernetes.io/projected/8ec0ad61-2e29-4aeb-8f2b-00db33de0861-kube-api-access-7h87w\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s\" (UID: \"8ec0ad61-2e29-4aeb-8f2b-00db33de0861\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"
Aug 04 18:19:01.203149 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:01.203094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ec0ad61-2e29-4aeb-8f2b-00db33de0861-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s\" (UID: \"8ec0ad61-2e29-4aeb-8f2b-00db33de0861\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"
Aug 04 18:19:01.205598 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:01.205566    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ec0ad61-2e29-4aeb-8f2b-00db33de0861-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s\" (UID: \"8ec0ad61-2e29-4aeb-8f2b-00db33de0861\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"
Aug 04 18:19:01.211738 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:01.211712    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h87w\" (UniqueName: \"kubernetes.io/projected/8ec0ad61-2e29-4aeb-8f2b-00db33de0861-kube-api-access-7h87w\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s\" (UID: \"8ec0ad61-2e29-4aeb-8f2b-00db33de0861\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"
Aug 04 18:19:01.260775 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:01.260739    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"
Aug 04 18:19:01.397817 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:01.397792    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"]
Aug 04 18:19:01.399901 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:19:01.399871    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec0ad61_2e29_4aeb_8f2b_00db33de0861.slice/crio-9818f5917157688104325b7c7f70182a0d5a96a43cf839b39bcae3b955043c9d WatchSource:0}: Error finding container 9818f5917157688104325b7c7f70182a0d5a96a43cf839b39bcae3b955043c9d: Status 404 returned error can't find the container with id 9818f5917157688104325b7c7f70182a0d5a96a43cf839b39bcae3b955043c9d
Aug 04 18:19:01.809426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:01.809390    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s" event={"ID":"8ec0ad61-2e29-4aeb-8f2b-00db33de0861","Type":"ContainerStarted","Data":"31d29a8851cc21b59d6190e8aac9bb1e76fa333214df21f20401510fa7916792"}
Aug 04 18:19:01.809858 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:01.809470    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s" event={"ID":"8ec0ad61-2e29-4aeb-8f2b-00db33de0861","Type":"ContainerStarted","Data":"9818f5917157688104325b7c7f70182a0d5a96a43cf839b39bcae3b955043c9d"}
Aug 04 18:19:01.809858 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:01.809529    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"
Aug 04 18:19:01.829704 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:01.829641    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s" podStartSLOduration=1.8296259350000001 podStartE2EDuration="1.829625935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:01.8275779 +0000 UTC m=+1141.803409614" watchObservedRunningTime="2026-08-04 18:19:01.829625935 +0000 UTC m=+1141.805457631"
Aug 04 18:19:07.694754 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:07.694721    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"
Aug 04 18:19:13.411899 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:13.411863    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:32.816150 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:32.816044    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"
Aug 04 18:19:34.514253 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:34.514215    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"]
Aug 04 18:19:34.519164 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:34.519131    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:34.620042 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:34.620012    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:42.816634 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:42.816590    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s" podUID="8ec0ad61-2e29-4aeb-8f2b-00db33de0861" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:19:45.687336 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:45.687292    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s" message="" err="command '/bin/sleep 15' exited with 137: "
Aug 04 18:19:45.687336 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:19:45.687334    2576 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s" podUID="8ec0ad61-2e29-4aeb-8f2b-00db33de0861" containerName="main" containerID="cri-o://31d29a8851cc21b59d6190e8aac9bb1e76fa333214df21f20401510fa7916792"
Aug 04 18:19:45.687789 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:45.687384    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s" podUID="8ec0ad61-2e29-4aeb-8f2b-00db33de0861" containerName="main" containerID="cri-o://31d29a8851cc21b59d6190e8aac9bb1e76fa333214df21f20401510fa7916792" gracePeriod=49
Aug 04 18:19:45.835227 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:45.835203    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s_8ec0ad61-2e29-4aeb-8f2b-00db33de0861/main/0.log"
Aug 04 18:19:45.837479 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:45.837463    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"
Aug 04 18:19:45.898057 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:45.898025    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ec0ad61-2e29-4aeb-8f2b-00db33de0861-tls-certs\") pod \"8ec0ad61-2e29-4aeb-8f2b-00db33de0861\" (UID: \"8ec0ad61-2e29-4aeb-8f2b-00db33de0861\") "
Aug 04 18:19:45.898267 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:45.898135    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h87w\" (UniqueName: \"kubernetes.io/projected/8ec0ad61-2e29-4aeb-8f2b-00db33de0861-kube-api-access-7h87w\") pod \"8ec0ad61-2e29-4aeb-8f2b-00db33de0861\" (UID: \"8ec0ad61-2e29-4aeb-8f2b-00db33de0861\") "
Aug 04 18:19:45.900657 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:45.900627    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec0ad61-2e29-4aeb-8f2b-00db33de0861-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ec0ad61-2e29-4aeb-8f2b-00db33de0861" (UID: "8ec0ad61-2e29-4aeb-8f2b-00db33de0861"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:19:45.900657 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:45.900631    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec0ad61-2e29-4aeb-8f2b-00db33de0861-kube-api-access-7h87w" (OuterVolumeSpecName: "kube-api-access-7h87w") pod "8ec0ad61-2e29-4aeb-8f2b-00db33de0861" (UID: "8ec0ad61-2e29-4aeb-8f2b-00db33de0861"). InnerVolumeSpecName "kube-api-access-7h87w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:19:46.000091 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:46.000021    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ec0ad61-2e29-4aeb-8f2b-00db33de0861-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:19:46.000091 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:46.000044    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h87w\" (UniqueName: \"kubernetes.io/projected/8ec0ad61-2e29-4aeb-8f2b-00db33de0861-kube-api-access-7h87w\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:19:46.028951 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:46.028927    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s_8ec0ad61-2e29-4aeb-8f2b-00db33de0861/main/0.log"
Aug 04 18:19:46.029087 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:46.028968    2576 generic.go:356] "Generic (PLEG): container finished" podID="8ec0ad61-2e29-4aeb-8f2b-00db33de0861" containerID="31d29a8851cc21b59d6190e8aac9bb1e76fa333214df21f20401510fa7916792" exitCode=1
Aug 04 18:19:46.029087 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:46.029031    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"
Aug 04 18:19:46.029087 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:46.029045    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s" event={"ID":"8ec0ad61-2e29-4aeb-8f2b-00db33de0861","Type":"ContainerDied","Data":"31d29a8851cc21b59d6190e8aac9bb1e76fa333214df21f20401510fa7916792"}
Aug 04 18:19:46.029227 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:46.029107    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s" event={"ID":"8ec0ad61-2e29-4aeb-8f2b-00db33de0861","Type":"ContainerDied","Data":"9818f5917157688104325b7c7f70182a0d5a96a43cf839b39bcae3b955043c9d"}
Aug 04 18:19:46.029227 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:46.029133    2576 scope.go:117] "RemoveContainer" containerID="31d29a8851cc21b59d6190e8aac9bb1e76fa333214df21f20401510fa7916792"
Aug 04 18:19:46.038512 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:46.038493    2576 scope.go:117] "RemoveContainer" containerID="31d29a8851cc21b59d6190e8aac9bb1e76fa333214df21f20401510fa7916792"
Aug 04 18:19:46.038757 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:19:46.038740    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d29a8851cc21b59d6190e8aac9bb1e76fa333214df21f20401510fa7916792\": container with ID starting with 31d29a8851cc21b59d6190e8aac9bb1e76fa333214df21f20401510fa7916792 not found: ID does not exist" containerID="31d29a8851cc21b59d6190e8aac9bb1e76fa333214df21f20401510fa7916792"
Aug 04 18:19:46.038810 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:46.038763    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d29a8851cc21b59d6190e8aac9bb1e76fa333214df21f20401510fa7916792"} err="failed to get container status \"31d29a8851cc21b59d6190e8aac9bb1e76fa333214df21f20401510fa7916792\": rpc error: code = NotFound desc = could not find container \"31d29a8851cc21b59d6190e8aac9bb1e76fa333214df21f20401510fa7916792\": container with ID starting with 31d29a8851cc21b59d6190e8aac9bb1e76fa333214df21f20401510fa7916792 not found: ID does not exist"
Aug 04 18:19:46.050595 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:46.050573    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"]
Aug 04 18:19:46.053954 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:46.053934    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf7bh4s"]
Aug 04 18:19:46.581698 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:46.581658    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ec0ad61-2e29-4aeb-8f2b-00db33de0861" path="/var/lib/kubelet/pods/8ec0ad61-2e29-4aeb-8f2b-00db33de0861/volumes"
Aug 04 18:19:54.553800 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:54.553761    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:54.650988 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:54.650951    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:57.735372 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:57.735340    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:57.924179 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:57.924142    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"]
Aug 04 18:19:57.924561 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:57.924544    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec0ad61-2e29-4aeb-8f2b-00db33de0861" containerName="main"
Aug 04 18:19:57.924561 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:57.924563    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec0ad61-2e29-4aeb-8f2b-00db33de0861" containerName="main"
Aug 04 18:19:57.924724 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:57.924656    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec0ad61-2e29-4aeb-8f2b-00db33de0861" containerName="main"
Aug 04 18:19:57.927701 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:57.927679    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:57.931214 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:57.931189    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Aug 04 18:19:57.931214 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:57.931209    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Aug 04 18:19:57.931390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:57.931191    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Aug 04 18:19:57.931390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:57.931318    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-nt9t6\""
Aug 04 18:19:57.931515 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:57.931502    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-rflsd\""
Aug 04 18:19:57.937828 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:57.937808    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"]
Aug 04 18:19:57.950016 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:57.949986    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:58.007039 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:58.006947    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbwbv\" (UniqueName: \"kubernetes.io/projected/faab4806-7808-460b-b233-876914ec4a7f-kube-api-access-bbwbv\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795zlns6\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:58.007039 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:58.006991    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faab4806-7808-460b-b233-876914ec4a7f-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795zlns6\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:58.007287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:58.007062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/faab4806-7808-460b-b233-876914ec4a7f-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795zlns6\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:58.007287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:58.007088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/faab4806-7808-460b-b233-876914ec4a7f-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795zlns6\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:58.107971 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:58.107936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/faab4806-7808-460b-b233-876914ec4a7f-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795zlns6\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:58.107971 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:58.107977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/faab4806-7808-460b-b233-876914ec4a7f-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795zlns6\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:58.108231 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:58.108040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbwbv\" (UniqueName: \"kubernetes.io/projected/faab4806-7808-460b-b233-876914ec4a7f-kube-api-access-bbwbv\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795zlns6\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:58.108231 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:58.108066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faab4806-7808-460b-b233-876914ec4a7f-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795zlns6\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:58.108359 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:58.108340    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/faab4806-7808-460b-b233-876914ec4a7f-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795zlns6\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:58.108415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:58.108394    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/faab4806-7808-460b-b233-876914ec4a7f-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795zlns6\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:58.110476 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:58.110456    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faab4806-7808-460b-b233-876914ec4a7f-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795zlns6\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:58.117548 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:58.117520    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbwbv\" (UniqueName: \"kubernetes.io/projected/faab4806-7808-460b-b233-876914ec4a7f-kube-api-access-bbwbv\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795zlns6\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:58.239090 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:58.239045    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:58.377337 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:58.377313    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"]
Aug 04 18:19:58.379435 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:19:58.379402    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaab4806_7808_460b_b233_876914ec4a7f.slice/crio-b590652a57f7e9771265348b0ad5840fcac81eb2abec5ed9ceb471f2a764a27f WatchSource:0}: Error finding container b590652a57f7e9771265348b0ad5840fcac81eb2abec5ed9ceb471f2a764a27f: Status 404 returned error can't find the container with id b590652a57f7e9771265348b0ad5840fcac81eb2abec5ed9ceb471f2a764a27f
Aug 04 18:19:59.093503 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:59.093460    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6" event={"ID":"faab4806-7808-460b-b233-876914ec4a7f","Type":"ContainerStarted","Data":"693efddb445f77f3ddb60f43c2a81d5db5d97b0560bd484186219c3a0813077d"}
Aug 04 18:19:59.093896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:59.093540    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:19:59.093896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:59.093569    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6" event={"ID":"faab4806-7808-460b-b233-876914ec4a7f","Type":"ContainerStarted","Data":"b590652a57f7e9771265348b0ad5840fcac81eb2abec5ed9ceb471f2a764a27f"}
Aug 04 18:19:59.111985 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:19:59.111827    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6" podStartSLOduration=2.111808053 podStartE2EDuration="2.111808053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:59.111219533 +0000 UTC m=+1199.087051222" watchObservedRunningTime="2026-08-04 18:19:59.111808053 +0000 UTC m=+1199.087639752"
Aug 04 18:20:01.180477 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:01.180446    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:06.779341 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:06.779300    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:06.879234 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:06.879199    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:11.107017 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:11.106976    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:30.101012 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:30.100979    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:20:31.374265 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:31.374224    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"]
Aug 04 18:20:31.386309 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:31.386275    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:31.448848 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:31.448815    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:37.525902 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:37.525869    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"]
Aug 04 18:20:37.532788 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:37.532729    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:37.629822 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:37.629791    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:37.694446 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:37.694405    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx" podUID="c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:20:40.100706 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:40.100667    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6" podUID="faab4806-7808-460b-b233-876914ec4a7f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:20:46.394997 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:46.394951    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6" podUID="faab4806-7808-460b-b233-876914ec4a7f" containerName="main" containerID="cri-o://693efddb445f77f3ddb60f43c2a81d5db5d97b0560bd484186219c3a0813077d" gracePeriod=45
Aug 04 18:20:46.651156 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:46.651087    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:20:46.741323 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:46.741292    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/faab4806-7808-460b-b233-876914ec4a7f-tokenizer-cache\") pod \"faab4806-7808-460b-b233-876914ec4a7f\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") "
Aug 04 18:20:46.741491 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:46.741387    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbwbv\" (UniqueName: \"kubernetes.io/projected/faab4806-7808-460b-b233-876914ec4a7f-kube-api-access-bbwbv\") pod \"faab4806-7808-460b-b233-876914ec4a7f\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") "
Aug 04 18:20:46.741491 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:46.741407    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/faab4806-7808-460b-b233-876914ec4a7f-cachi2-cache\") pod \"faab4806-7808-460b-b233-876914ec4a7f\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") "
Aug 04 18:20:46.741491 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:46.741453    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faab4806-7808-460b-b233-876914ec4a7f-tls-certs\") pod \"faab4806-7808-460b-b233-876914ec4a7f\" (UID: \"faab4806-7808-460b-b233-876914ec4a7f\") "
Aug 04 18:20:46.741622 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:46.741549    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faab4806-7808-460b-b233-876914ec4a7f-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "faab4806-7808-460b-b233-876914ec4a7f" (UID: "faab4806-7808-460b-b233-876914ec4a7f"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:20:46.741678 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:46.741636    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faab4806-7808-460b-b233-876914ec4a7f-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "faab4806-7808-460b-b233-876914ec4a7f" (UID: "faab4806-7808-460b-b233-876914ec4a7f"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:20:46.741737 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:46.741721    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/faab4806-7808-460b-b233-876914ec4a7f-tokenizer-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:20:46.741789 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:46.741739    2576 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/faab4806-7808-460b-b233-876914ec4a7f-cachi2-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:20:46.743539 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:46.743517    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faab4806-7808-460b-b233-876914ec4a7f-kube-api-access-bbwbv" (OuterVolumeSpecName: "kube-api-access-bbwbv") pod "faab4806-7808-460b-b233-876914ec4a7f" (UID: "faab4806-7808-460b-b233-876914ec4a7f"). InnerVolumeSpecName "kube-api-access-bbwbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:20:46.743657 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:46.743554    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faab4806-7808-460b-b233-876914ec4a7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "faab4806-7808-460b-b233-876914ec4a7f" (UID: "faab4806-7808-460b-b233-876914ec4a7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:20:46.842912 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:46.842876    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbwbv\" (UniqueName: \"kubernetes.io/projected/faab4806-7808-460b-b233-876914ec4a7f-kube-api-access-bbwbv\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:20:46.842912 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:46.842909    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faab4806-7808-460b-b233-876914ec4a7f-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:20:47.326969 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:47.326929    2576 generic.go:356] "Generic (PLEG): container finished" podID="faab4806-7808-460b-b233-876914ec4a7f" containerID="693efddb445f77f3ddb60f43c2a81d5db5d97b0560bd484186219c3a0813077d" exitCode=0
Aug 04 18:20:47.327187 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:47.326972    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"
Aug 04 18:20:47.327187 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:47.326981    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6" event={"ID":"faab4806-7808-460b-b233-876914ec4a7f","Type":"ContainerDied","Data":"693efddb445f77f3ddb60f43c2a81d5db5d97b0560bd484186219c3a0813077d"}
Aug 04 18:20:47.327187 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:47.327036    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6" event={"ID":"faab4806-7808-460b-b233-876914ec4a7f","Type":"ContainerDied","Data":"b590652a57f7e9771265348b0ad5840fcac81eb2abec5ed9ceb471f2a764a27f"}
Aug 04 18:20:47.327187 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:47.327059    2576 scope.go:117] "RemoveContainer" containerID="693efddb445f77f3ddb60f43c2a81d5db5d97b0560bd484186219c3a0813077d"
Aug 04 18:20:47.338456 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:47.338433    2576 scope.go:117] "RemoveContainer" containerID="693efddb445f77f3ddb60f43c2a81d5db5d97b0560bd484186219c3a0813077d"
Aug 04 18:20:47.338779 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:20:47.338751    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693efddb445f77f3ddb60f43c2a81d5db5d97b0560bd484186219c3a0813077d\": container with ID starting with 693efddb445f77f3ddb60f43c2a81d5db5d97b0560bd484186219c3a0813077d not found: ID does not exist" containerID="693efddb445f77f3ddb60f43c2a81d5db5d97b0560bd484186219c3a0813077d"
Aug 04 18:20:47.338852 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:47.338792    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693efddb445f77f3ddb60f43c2a81d5db5d97b0560bd484186219c3a0813077d"} err="failed to get container status \"693efddb445f77f3ddb60f43c2a81d5db5d97b0560bd484186219c3a0813077d\": rpc error: code = NotFound desc = could not find container \"693efddb445f77f3ddb60f43c2a81d5db5d97b0560bd484186219c3a0813077d\": container with ID starting with 693efddb445f77f3ddb60f43c2a81d5db5d97b0560bd484186219c3a0813077d not found: ID does not exist"
Aug 04 18:20:47.357230 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:47.357201    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"]
Aug 04 18:20:47.362048 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:47.362024    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795zlns6"]
Aug 04 18:20:47.694308 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:47.694224    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx" podUID="c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:20:48.581979 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:48.581943    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faab4806-7808-460b-b233-876914ec4a7f" path="/var/lib/kubelet/pods/faab4806-7808-460b-b233-876914ec4a7f/volumes"
Aug 04 18:20:51.416032 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:51.416000    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:51.510180 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:51.510147    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:52.547574 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:52.547487    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx" podUID="c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3" containerName="main" containerID="cri-o://8467053ef044f66cae54eef6d15fc160baa4476181f23af418562f3d20b9e30b" gracePeriod=45
Aug 04 18:20:52.797321 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:52.797292    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"
Aug 04 18:20:52.902242 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:52.902163    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3-tls-certs\") pod \"c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3\" (UID: \"c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3\") "
Aug 04 18:20:52.902242 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:52.902210    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpc4c\" (UniqueName: \"kubernetes.io/projected/c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3-kube-api-access-hpc4c\") pod \"c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3\" (UID: \"c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3\") "
Aug 04 18:20:52.904375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:52.904333    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3" (UID: "c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:20:52.904487 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:52.904431    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3-kube-api-access-hpc4c" (OuterVolumeSpecName: "kube-api-access-hpc4c") pod "c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3" (UID: "c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3"). InnerVolumeSpecName "kube-api-access-hpc4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:20:53.002809 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:53.002774    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:20:53.002809 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:53.002803    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpc4c\" (UniqueName: \"kubernetes.io/projected/c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3-kube-api-access-hpc4c\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:20:53.357551 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:53.357520    2576 generic.go:356] "Generic (PLEG): container finished" podID="c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3" containerID="8467053ef044f66cae54eef6d15fc160baa4476181f23af418562f3d20b9e30b" exitCode=0
Aug 04 18:20:53.357726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:53.357560    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"
Aug 04 18:20:53.357726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:53.357569    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx" event={"ID":"c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3","Type":"ContainerDied","Data":"8467053ef044f66cae54eef6d15fc160baa4476181f23af418562f3d20b9e30b"}
Aug 04 18:20:53.357726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:53.357610    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx" event={"ID":"c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3","Type":"ContainerDied","Data":"25c3e77591d40620b60b68c3584e49d468f74b9eb1f1f55c9e3b122dbb1308d6"}
Aug 04 18:20:53.357726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:53.357631    2576 scope.go:117] "RemoveContainer" containerID="8467053ef044f66cae54eef6d15fc160baa4476181f23af418562f3d20b9e30b"
Aug 04 18:20:53.366944 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:53.366926    2576 scope.go:117] "RemoveContainer" containerID="8467053ef044f66cae54eef6d15fc160baa4476181f23af418562f3d20b9e30b"
Aug 04 18:20:53.367216 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:20:53.367198    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8467053ef044f66cae54eef6d15fc160baa4476181f23af418562f3d20b9e30b\": container with ID starting with 8467053ef044f66cae54eef6d15fc160baa4476181f23af418562f3d20b9e30b not found: ID does not exist" containerID="8467053ef044f66cae54eef6d15fc160baa4476181f23af418562f3d20b9e30b"
Aug 04 18:20:53.367273 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:53.367224    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8467053ef044f66cae54eef6d15fc160baa4476181f23af418562f3d20b9e30b"} err="failed to get container status \"8467053ef044f66cae54eef6d15fc160baa4476181f23af418562f3d20b9e30b\": rpc error: code = NotFound desc = could not find container \"8467053ef044f66cae54eef6d15fc160baa4476181f23af418562f3d20b9e30b\": container with ID starting with 8467053ef044f66cae54eef6d15fc160baa4476181f23af418562f3d20b9e30b not found: ID does not exist"
Aug 04 18:20:53.381543 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:53.380889    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"]
Aug 04 18:20:53.385683 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:53.385660    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dckvnqx"]
Aug 04 18:20:54.581420 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:54.581386    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3" path="/var/lib/kubelet/pods/c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3/volumes"
Aug 04 18:20:57.483673 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:57.483642    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:58.795485 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:20:58.795452    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:02.517956 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:02.517919    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:02.574963 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:02.574930    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:03.925724 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:03.925688    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:04.843967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.843926    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"]
Aug 04 18:21:04.844644 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.844622    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faab4806-7808-460b-b233-876914ec4a7f" containerName="main"
Aug 04 18:21:04.844724 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.844650    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="faab4806-7808-460b-b233-876914ec4a7f" containerName="main"
Aug 04 18:21:04.844724 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.844674    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3" containerName="main"
Aug 04 18:21:04.844724 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.844682    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3" containerName="main"
Aug 04 18:21:04.844825 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.844770    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c83b83f7-a1ae-4806-b5cf-fd9e1c89d5b3" containerName="main"
Aug 04 18:21:04.844825 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.844786    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="faab4806-7808-460b-b233-876914ec4a7f" containerName="main"
Aug 04 18:21:04.849639 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.849622    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:04.852517 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.852486    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Aug 04 18:21:04.853379 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.853359    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-smng9\""
Aug 04 18:21:04.853487 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.853401    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Aug 04 18:21:04.853487 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.853447    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-xdbl6\""
Aug 04 18:21:04.853596 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.853488    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Aug 04 18:21:04.857716 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.857448    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"]
Aug 04 18:21:04.877101 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.877075    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:04.910660 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.910628    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc5xg\" (UniqueName: \"kubernetes.io/projected/a42de834-938b-4f65-94f6-831dc6af5faa-kube-api-access-wc5xg\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:04.910796 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.910666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:04.910796 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.910687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a42de834-938b-4f65-94f6-831dc6af5faa-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:04.910796 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.910763    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:04.910796 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.910792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:04.910932 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:04.910818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:05.011925 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.011884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc5xg\" (UniqueName: \"kubernetes.io/projected/a42de834-938b-4f65-94f6-831dc6af5faa-kube-api-access-wc5xg\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:05.012450 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.011936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:05.012450 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.011963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a42de834-938b-4f65-94f6-831dc6af5faa-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:05.012450 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.012016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:05.012450 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.012044    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:05.012450 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.012084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:05.012450 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.012332    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:05.012450 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.012372    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:05.012781 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.012471    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:05.014334 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.014316    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:05.014517 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.014501    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a42de834-938b-4f65-94f6-831dc6af5faa-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:05.020800 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.020780    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc5xg\" (UniqueName: \"kubernetes.io/projected/a42de834-938b-4f65-94f6-831dc6af5faa-kube-api-access-wc5xg\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:05.161203 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.161098    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:05.255384 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.255336    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:05.299576 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.299550    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"]
Aug 04 18:21:05.301727 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:21:05.301698    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42de834_938b_4f65_94f6_831dc6af5faa.slice/crio-88b2adf9bb7e2d0b1da850dec84acbf07abfb34694bd9a7b499bb6feaaf13edd WatchSource:0}: Error finding container 88b2adf9bb7e2d0b1da850dec84acbf07abfb34694bd9a7b499bb6feaaf13edd: Status 404 returned error can't find the container with id 88b2adf9bb7e2d0b1da850dec84acbf07abfb34694bd9a7b499bb6feaaf13edd
Aug 04 18:21:05.304125 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.304092    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:21:05.412478 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:05.412397    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p" event={"ID":"a42de834-938b-4f65-94f6-831dc6af5faa","Type":"ContainerStarted","Data":"88b2adf9bb7e2d0b1da850dec84acbf07abfb34694bd9a7b499bb6feaaf13edd"}
Aug 04 18:21:07.422998 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:07.422907    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p" event={"ID":"a42de834-938b-4f65-94f6-831dc6af5faa","Type":"ContainerStarted","Data":"77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976"}
Aug 04 18:21:07.423437 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:07.423005    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:08.429816 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:08.429782    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p" event={"ID":"a42de834-938b-4f65-94f6-831dc6af5faa","Type":"ContainerStarted","Data":"cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0"}
Aug 04 18:21:08.449544 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:08.449497    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p" podStartSLOduration=2.689955509 podStartE2EDuration="4.449481183s" totalImagesPullingTime="1.759525674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:21:08.447621768 +0000 UTC m=+1268.423453475" watchObservedRunningTime="2026-08-04 18:21:08.449481183 +0000 UTC m=+1268.425312917"
Aug 04 18:21:09.161648 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:09.161616    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:09.414056 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:09.413913    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:09.475917 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:09.475882    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:15.161518 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:15.161476    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:15.173892 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:15.173861    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:15.478225 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:15.478150    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:20.338084 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.338047    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:20.497053 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.497015    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"]
Aug 04 18:21:20.500808 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.500785    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"
Aug 04 18:21:20.504399 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.504210    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Aug 04 18:21:20.504399 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.504221    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-zmlq7\""
Aug 04 18:21:20.504399 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.504221    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-d9wnb\""
Aug 04 18:21:20.504399 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.504272    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Aug 04 18:21:20.504399 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.504309    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Aug 04 18:21:20.512316 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.512291    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"]
Aug 04 18:21:20.534833 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.534789    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:20.552844 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.552813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d982bf2-56f5-4999-b05c-9668afda9d40-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm\" (UID: \"3d982bf2-56f5-4999-b05c-9668afda9d40\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"
Aug 04 18:21:20.552968 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.552869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppls4\" (UniqueName: \"kubernetes.io/projected/3d982bf2-56f5-4999-b05c-9668afda9d40-kube-api-access-ppls4\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm\" (UID: \"3d982bf2-56f5-4999-b05c-9668afda9d40\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"
Aug 04 18:21:20.653512 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.653429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d982bf2-56f5-4999-b05c-9668afda9d40-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm\" (UID: \"3d982bf2-56f5-4999-b05c-9668afda9d40\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"
Aug 04 18:21:20.653512 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.653493    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppls4\" (UniqueName: \"kubernetes.io/projected/3d982bf2-56f5-4999-b05c-9668afda9d40-kube-api-access-ppls4\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm\" (UID: \"3d982bf2-56f5-4999-b05c-9668afda9d40\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"
Aug 04 18:21:20.655902 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.655877    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d982bf2-56f5-4999-b05c-9668afda9d40-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm\" (UID: \"3d982bf2-56f5-4999-b05c-9668afda9d40\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"
Aug 04 18:21:20.662264 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.662238    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppls4\" (UniqueName: \"kubernetes.io/projected/3d982bf2-56f5-4999-b05c-9668afda9d40-kube-api-access-ppls4\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm\" (UID: \"3d982bf2-56f5-4999-b05c-9668afda9d40\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"
Aug 04 18:21:20.814168 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.814138    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"
Aug 04 18:21:20.947265 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:20.947241    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"]
Aug 04 18:21:20.949406 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:21:20.949376    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d982bf2_56f5_4999_b05c_9668afda9d40.slice/crio-1dd33e8e3000f172b304571abfcd189aedec968623d58146a0f744b126fe3401 WatchSource:0}: Error finding container 1dd33e8e3000f172b304571abfcd189aedec968623d58146a0f744b126fe3401: Status 404 returned error can't find the container with id 1dd33e8e3000f172b304571abfcd189aedec968623d58146a0f744b126fe3401
Aug 04 18:21:21.498015 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:21.497970    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm" event={"ID":"3d982bf2-56f5-4999-b05c-9668afda9d40","Type":"ContainerStarted","Data":"fd1a719106059af1150e84229df22c16bfef42ba579699f3b9326693a0a3be0c"}
Aug 04 18:21:21.498498 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:21.498039    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm" event={"ID":"3d982bf2-56f5-4999-b05c-9668afda9d40","Type":"ContainerStarted","Data":"1dd33e8e3000f172b304571abfcd189aedec968623d58146a0f744b126fe3401"}
Aug 04 18:21:21.498498 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:21.498063    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"
Aug 04 18:21:21.516800 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:21.516747    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm" podStartSLOduration=1.5167306520000001 podStartE2EDuration="1.516730652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:21:21.516152344 +0000 UTC m=+1281.491984071" watchObservedRunningTime="2026-08-04 18:21:21.516730652 +0000 UTC m=+1281.492562348"
Aug 04 18:21:26.484364 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:26.484335    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:21:29.966224 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:29.966175    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:39.388048 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:39.388012    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:52.504908 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:21:52.504879    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"
Aug 04 18:22:43.111743 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.111709    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:43.189116 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.189073    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"]
Aug 04 18:22:43.198169 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:22:43.198125    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:22:43.198169 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:22:43.198169 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:22:43.198169 ip-10-0-133-187 kubenswrapper[2576]:  > containerID="cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0" cmd=["/bin/sleep","15"]
Aug 04 18:22:43.198463 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.198185    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p" message="" err=<
Aug 04 18:22:43.198463 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:22:43.198463 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:22:43.198463 ip-10-0-133-187 kubenswrapper[2576]:  >
Aug 04 18:22:43.198463 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:22:43.198200    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:22:43.198463 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:22:43.198463 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:22:43.198463 ip-10-0-133-187 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p" podUID="a42de834-938b-4f65-94f6-831dc6af5faa" containerName="main" containerID="cri-o://cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0"
Aug 04 18:22:43.198463 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.198254    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p" podUID="a42de834-938b-4f65-94f6-831dc6af5faa" containerName="main" containerID="cri-o://cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0" gracePeriod=60
Aug 04 18:22:43.211672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.211645    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:43.413373 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.413337    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p" podUID="a42de834-938b-4f65-94f6-831dc6af5faa" containerName="llm-d-routing-sidecar" containerID="cri-o://77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976" gracePeriod=60
Aug 04 18:22:43.467782 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.467738    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p" podUID="a42de834-938b-4f65-94f6-831dc6af5faa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8001/health\": dial tcp 10.134.0.45:8001: connect: connection refused"
Aug 04 18:22:43.660834 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.660805    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:22:43.697387 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.697310    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-model-cache\") pod \"a42de834-938b-4f65-94f6-831dc6af5faa\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") "
Aug 04 18:22:43.697387 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.697363    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-dshm\") pod \"a42de834-938b-4f65-94f6-831dc6af5faa\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") "
Aug 04 18:22:43.697387 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.697386    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-home\") pod \"a42de834-938b-4f65-94f6-831dc6af5faa\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") "
Aug 04 18:22:43.697658 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.697411    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a42de834-938b-4f65-94f6-831dc6af5faa-tls-certs\") pod \"a42de834-938b-4f65-94f6-831dc6af5faa\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") "
Aug 04 18:22:43.697658 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.697499    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-tmp-dir\") pod \"a42de834-938b-4f65-94f6-831dc6af5faa\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") "
Aug 04 18:22:43.697658 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.697588    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc5xg\" (UniqueName: \"kubernetes.io/projected/a42de834-938b-4f65-94f6-831dc6af5faa-kube-api-access-wc5xg\") pod \"a42de834-938b-4f65-94f6-831dc6af5faa\" (UID: \"a42de834-938b-4f65-94f6-831dc6af5faa\") "
Aug 04 18:22:43.697822 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.697652    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-model-cache" (OuterVolumeSpecName: "model-cache") pod "a42de834-938b-4f65-94f6-831dc6af5faa" (UID: "a42de834-938b-4f65-94f6-831dc6af5faa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:22:43.697822 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.697662    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-home" (OuterVolumeSpecName: "home") pod "a42de834-938b-4f65-94f6-831dc6af5faa" (UID: "a42de834-938b-4f65-94f6-831dc6af5faa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:22:43.697822 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.697700    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a42de834-938b-4f65-94f6-831dc6af5faa" (UID: "a42de834-938b-4f65-94f6-831dc6af5faa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:22:43.697958 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.697902    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:22:43.697958 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.697923    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:22:43.697958 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.697936    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:22:43.699711 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.699686    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42de834-938b-4f65-94f6-831dc6af5faa-kube-api-access-wc5xg" (OuterVolumeSpecName: "kube-api-access-wc5xg") pod "a42de834-938b-4f65-94f6-831dc6af5faa" (UID: "a42de834-938b-4f65-94f6-831dc6af5faa"). InnerVolumeSpecName "kube-api-access-wc5xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:22:43.700020 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.700004    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a42de834-938b-4f65-94f6-831dc6af5faa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a42de834-938b-4f65-94f6-831dc6af5faa" (UID: "a42de834-938b-4f65-94f6-831dc6af5faa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:22:43.700070 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.700020    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-dshm" (OuterVolumeSpecName: "dshm") pod "a42de834-938b-4f65-94f6-831dc6af5faa" (UID: "a42de834-938b-4f65-94f6-831dc6af5faa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:22:43.798895 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.798861    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a42de834-938b-4f65-94f6-831dc6af5faa-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:22:43.798895 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.798891    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a42de834-938b-4f65-94f6-831dc6af5faa-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:22:43.798895 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.798902    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc5xg\" (UniqueName: \"kubernetes.io/projected/a42de834-938b-4f65-94f6-831dc6af5faa-kube-api-access-wc5xg\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:22:43.910487 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.910444    2576 generic.go:356] "Generic (PLEG): container finished" podID="a42de834-938b-4f65-94f6-831dc6af5faa" containerID="cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0" exitCode=0
Aug 04 18:22:43.910487 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.910474    2576 generic.go:356] "Generic (PLEG): container finished" podID="a42de834-938b-4f65-94f6-831dc6af5faa" containerID="77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976" exitCode=0
Aug 04 18:22:43.910708 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.910498    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"
Aug 04 18:22:43.910708 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.910531    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p" event={"ID":"a42de834-938b-4f65-94f6-831dc6af5faa","Type":"ContainerDied","Data":"cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0"}
Aug 04 18:22:43.910708 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.910591    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p" event={"ID":"a42de834-938b-4f65-94f6-831dc6af5faa","Type":"ContainerDied","Data":"77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976"}
Aug 04 18:22:43.910708 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.910609    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p" event={"ID":"a42de834-938b-4f65-94f6-831dc6af5faa","Type":"ContainerDied","Data":"88b2adf9bb7e2d0b1da850dec84acbf07abfb34694bd9a7b499bb6feaaf13edd"}
Aug 04 18:22:43.910708 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.910630    2576 scope.go:117] "RemoveContainer" containerID="cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0"
Aug 04 18:22:43.919348 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.919331    2576 scope.go:117] "RemoveContainer" containerID="77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976"
Aug 04 18:22:43.926974 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.926946    2576 scope.go:117] "RemoveContainer" containerID="cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0"
Aug 04 18:22:43.927252 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:22:43.927230    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0\": container with ID starting with cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0 not found: ID does not exist" containerID="cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0"
Aug 04 18:22:43.927311 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.927260    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0"} err="failed to get container status \"cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0\": rpc error: code = NotFound desc = could not find container \"cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0\": container with ID starting with cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0 not found: ID does not exist"
Aug 04 18:22:43.927311 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.927277    2576 scope.go:117] "RemoveContainer" containerID="77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976"
Aug 04 18:22:43.927507 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:22:43.927490    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976\": container with ID starting with 77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976 not found: ID does not exist" containerID="77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976"
Aug 04 18:22:43.927545 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.927513    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976"} err="failed to get container status \"77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976\": rpc error: code = NotFound desc = could not find container \"77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976\": container with ID starting with 77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976 not found: ID does not exist"
Aug 04 18:22:43.927545 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.927531    2576 scope.go:117] "RemoveContainer" containerID="cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0"
Aug 04 18:22:43.927750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.927731    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0"} err="failed to get container status \"cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0\": rpc error: code = NotFound desc = could not find container \"cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0\": container with ID starting with cfaa71703cfc1ac7cdebccc0e5a92f11f6971dec99ec3f05a5b89bae549441d0 not found: ID does not exist"
Aug 04 18:22:43.927791 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.927751    2576 scope.go:117] "RemoveContainer" containerID="77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976"
Aug 04 18:22:43.927980 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.927960    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976"} err="failed to get container status \"77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976\": rpc error: code = NotFound desc = could not find container \"77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976\": container with ID starting with 77b32dc343997128613fa83dfff363c5e769deddbe6375775a0468d65f764976 not found: ID does not exist"
Aug 04 18:22:43.932443 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.932420    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"]
Aug 04 18:22:43.935694 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:43.935673    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-bxt8p"]
Aug 04 18:22:44.582435 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:44.582404    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a42de834-938b-4f65-94f6-831dc6af5faa" path="/var/lib/kubelet/pods/a42de834-938b-4f65-94f6-831dc6af5faa/volumes"
Aug 04 18:22:53.088698 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:22:53.088665    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:02.815981 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:02.815942    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:02.915623 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:02.915582    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:09.043948 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:09.043905    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:14.594168 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:14.594133    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:14.692521 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:14.692482    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:17.509564 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:17.509534    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:17.697384 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:17.697349    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:24.742820 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:24.742778    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:38.205964 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:38.205925    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"]
Aug 04 18:23:38.209265 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:38.209224    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:38.307260 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:38.307225    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:42.504720 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:42.504678    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm" podUID="3d982bf2-56f5-4999-b05c-9668afda9d40" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:23:51.735137 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:51.735082    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:51.801092 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:51.801053    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:52.505370 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:52.505283    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm" podUID="3d982bf2-56f5-4999-b05c-9668afda9d40" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:23:53.227321 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:53.227277    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm" podUID="3d982bf2-56f5-4999-b05c-9668afda9d40" containerName="main" containerID="cri-o://fd1a719106059af1150e84229df22c16bfef42ba579699f3b9326693a0a3be0c" gracePeriod=45
Aug 04 18:23:53.481620 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:53.481556    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"
Aug 04 18:23:53.640355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:53.640316    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d982bf2-56f5-4999-b05c-9668afda9d40-tls-certs\") pod \"3d982bf2-56f5-4999-b05c-9668afda9d40\" (UID: \"3d982bf2-56f5-4999-b05c-9668afda9d40\") "
Aug 04 18:23:53.640355 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:53.640364    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppls4\" (UniqueName: \"kubernetes.io/projected/3d982bf2-56f5-4999-b05c-9668afda9d40-kube-api-access-ppls4\") pod \"3d982bf2-56f5-4999-b05c-9668afda9d40\" (UID: \"3d982bf2-56f5-4999-b05c-9668afda9d40\") "
Aug 04 18:23:53.642552 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:53.642519    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d982bf2-56f5-4999-b05c-9668afda9d40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d982bf2-56f5-4999-b05c-9668afda9d40" (UID: "3d982bf2-56f5-4999-b05c-9668afda9d40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:23:53.642678 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:53.642568    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d982bf2-56f5-4999-b05c-9668afda9d40-kube-api-access-ppls4" (OuterVolumeSpecName: "kube-api-access-ppls4") pod "3d982bf2-56f5-4999-b05c-9668afda9d40" (UID: "3d982bf2-56f5-4999-b05c-9668afda9d40"). InnerVolumeSpecName "kube-api-access-ppls4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:23:53.741048 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:53.740966    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d982bf2-56f5-4999-b05c-9668afda9d40-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:23:53.741048 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:53.740993    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppls4\" (UniqueName: \"kubernetes.io/projected/3d982bf2-56f5-4999-b05c-9668afda9d40-kube-api-access-ppls4\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:23:54.252634 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:54.252595    2576 generic.go:356] "Generic (PLEG): container finished" podID="3d982bf2-56f5-4999-b05c-9668afda9d40" containerID="fd1a719106059af1150e84229df22c16bfef42ba579699f3b9326693a0a3be0c" exitCode=0
Aug 04 18:23:54.252634 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:54.252636    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"
Aug 04 18:23:54.253104 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:54.252668    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm" event={"ID":"3d982bf2-56f5-4999-b05c-9668afda9d40","Type":"ContainerDied","Data":"fd1a719106059af1150e84229df22c16bfef42ba579699f3b9326693a0a3be0c"}
Aug 04 18:23:54.253104 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:54.252714    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm" event={"ID":"3d982bf2-56f5-4999-b05c-9668afda9d40","Type":"ContainerDied","Data":"1dd33e8e3000f172b304571abfcd189aedec968623d58146a0f744b126fe3401"}
Aug 04 18:23:54.253104 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:54.252737    2576 scope.go:117] "RemoveContainer" containerID="fd1a719106059af1150e84229df22c16bfef42ba579699f3b9326693a0a3be0c"
Aug 04 18:23:54.261390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:54.261369    2576 scope.go:117] "RemoveContainer" containerID="fd1a719106059af1150e84229df22c16bfef42ba579699f3b9326693a0a3be0c"
Aug 04 18:23:54.261688 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:23:54.261671    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd1a719106059af1150e84229df22c16bfef42ba579699f3b9326693a0a3be0c\": container with ID starting with fd1a719106059af1150e84229df22c16bfef42ba579699f3b9326693a0a3be0c not found: ID does not exist" containerID="fd1a719106059af1150e84229df22c16bfef42ba579699f3b9326693a0a3be0c"
Aug 04 18:23:54.261755 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:54.261694    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd1a719106059af1150e84229df22c16bfef42ba579699f3b9326693a0a3be0c"} err="failed to get container status \"fd1a719106059af1150e84229df22c16bfef42ba579699f3b9326693a0a3be0c\": rpc error: code = NotFound desc = could not find container \"fd1a719106059af1150e84229df22c16bfef42ba579699f3b9326693a0a3be0c\": container with ID starting with fd1a719106059af1150e84229df22c16bfef42ba579699f3b9326693a0a3be0c not found: ID does not exist"
Aug 04 18:23:54.278217 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:54.278180    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"]
Aug 04 18:23:54.281232 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:54.281204    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-9k8vm"]
Aug 04 18:23:54.587726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:54.587689    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d982bf2-56f5-4999-b05c-9668afda9d40" path="/var/lib/kubelet/pods/3d982bf2-56f5-4999-b05c-9668afda9d40/volumes"
Aug 04 18:23:56.479024 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:56.478988    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:56.574450 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:56.574400    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:56.675758 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:23:56.675722    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:03.140912 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:03.140873    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:06.088681 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.088647    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"]
Aug 04 18:24:06.089238 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.089220    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a42de834-938b-4f65-94f6-831dc6af5faa" containerName="main"
Aug 04 18:24:06.089339 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.089240    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a42de834-938b-4f65-94f6-831dc6af5faa" containerName="main"
Aug 04 18:24:06.089339 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.089254    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d982bf2-56f5-4999-b05c-9668afda9d40" containerName="main"
Aug 04 18:24:06.089339 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.089262    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d982bf2-56f5-4999-b05c-9668afda9d40" containerName="main"
Aug 04 18:24:06.089339 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.089310    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a42de834-938b-4f65-94f6-831dc6af5faa" containerName="llm-d-routing-sidecar"
Aug 04 18:24:06.089339 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.089320    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a42de834-938b-4f65-94f6-831dc6af5faa" containerName="llm-d-routing-sidecar"
Aug 04 18:24:06.089587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.089433    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d982bf2-56f5-4999-b05c-9668afda9d40" containerName="main"
Aug 04 18:24:06.089587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.089447    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a42de834-938b-4f65-94f6-831dc6af5faa" containerName="main"
Aug 04 18:24:06.089587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.089460    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a42de834-938b-4f65-94f6-831dc6af5faa" containerName="llm-d-routing-sidecar"
Aug 04 18:24:06.092076 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.092055    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.094915 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.094890    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Aug 04 18:24:06.095044 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.094951    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Aug 04 18:24:06.095965 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.095934    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-fdcmj\""
Aug 04 18:24:06.096081 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.095968    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Aug 04 18:24:06.101336 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.101316    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"]
Aug 04 18:24:06.116349 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.116308    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:06.151800 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.151766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f9zb\" (UniqueName: \"kubernetes.io/projected/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-kube-api-access-5f9zb\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.151959 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.151808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-dshm\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.151959 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.151833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-home\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.151959 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.151916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.151959 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.151958    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.152194 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.151978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-model-cache\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.253580 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.253543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-dshm\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.253580 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.253586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-home\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.253855 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.253631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.253855 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.253660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.253855 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.253683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-model-cache\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.253855 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.253737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f9zb\" (UniqueName: \"kubernetes.io/projected/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-kube-api-access-5f9zb\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.254076 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.254054    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-home\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.254159 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.254077    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.254159 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.254147    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-model-cache\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.255894 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.255869    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-dshm\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.256146 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.256129    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.262494 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.262475    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f9zb\" (UniqueName: \"kubernetes.io/projected/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-kube-api-access-5f9zb\") pod \"canary-v1-kserve-65d58ccc9-j9rnv\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.403939 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.403851    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:06.572097 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:06.572068    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"]
Aug 04 18:24:06.574043 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:24:06.573907    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44cf1c8_52a9_4cdf_9210_7a0bf3fe97fb.slice/crio-3383b1af82b29deab919d2572729a403dd926f274ee8c0ff89c55686539dd0f1 WatchSource:0}: Error finding container 3383b1af82b29deab919d2572729a403dd926f274ee8c0ff89c55686539dd0f1: Status 404 returned error can't find the container with id 3383b1af82b29deab919d2572729a403dd926f274ee8c0ff89c55686539dd0f1
Aug 04 18:24:07.315912 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:07.315871    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv" event={"ID":"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb","Type":"ContainerStarted","Data":"3383b1af82b29deab919d2572729a403dd926f274ee8c0ff89c55686539dd0f1"}
Aug 04 18:24:08.324929 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:08.324893    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv" event={"ID":"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb","Type":"ContainerStarted","Data":"5112177dd88afddbadfe9d78b990fc096872c82d34647755433c93e915fd9043"}
Aug 04 18:24:08.342914 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:08.342864    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv" podStartSLOduration=1.394788648 podStartE2EDuration="2.342848036s" totalImagesPullingTime="948.059388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:24:08.341945895 +0000 UTC m=+1448.317777598" watchObservedRunningTime="2026-08-04 18:24:08.342848036 +0000 UTC m=+1448.318679733"
Aug 04 18:24:08.404543 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:08.404508    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:08.744842 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:08.744749    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:08.844672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:08.844636    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:11.675788 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:11.675741    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:11.764410 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:11.764381    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:16.404572 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:16.404536    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:16.417251 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:16.417222    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:17.378216 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:17.378185    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:24:18.576530 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:18.576493    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:19.889331 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:19.889283    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:23.368516 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:23.368456    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:26.952035 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:26.952002    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:27.047297 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:24:27.047256    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:25:14.845482 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:14.845437    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:25:15.194755 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.194669    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"]
Aug 04 18:25:15.198652 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.198635    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"
Aug 04 18:25:15.204385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.204361    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Aug 04 18:25:15.204523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.204419    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-hxlxn\""
Aug 04 18:25:15.204523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.204449    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-cjq6s\""
Aug 04 18:25:15.204523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.204460    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Aug 04 18:25:15.204700 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.204523    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Aug 04 18:25:15.219653 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.219631    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"]
Aug 04 18:25:15.224223 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.224080    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:25:15.293238 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.293206    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvlf\" (UniqueName: \"kubernetes.io/projected/7af3cc75-0e41-4799-90d5-3fa31f352a6a-kube-api-access-9zvlf\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw\" (UID: \"7af3cc75-0e41-4799-90d5-3fa31f352a6a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"
Aug 04 18:25:15.293417 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.293261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af3cc75-0e41-4799-90d5-3fa31f352a6a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw\" (UID: \"7af3cc75-0e41-4799-90d5-3fa31f352a6a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"
Aug 04 18:25:15.295633 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.295591    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:25:15.394476 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.394422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvlf\" (UniqueName: \"kubernetes.io/projected/7af3cc75-0e41-4799-90d5-3fa31f352a6a-kube-api-access-9zvlf\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw\" (UID: \"7af3cc75-0e41-4799-90d5-3fa31f352a6a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"
Aug 04 18:25:15.394655 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.394579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af3cc75-0e41-4799-90d5-3fa31f352a6a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw\" (UID: \"7af3cc75-0e41-4799-90d5-3fa31f352a6a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"
Aug 04 18:25:15.397007 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.396984    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af3cc75-0e41-4799-90d5-3fa31f352a6a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw\" (UID: \"7af3cc75-0e41-4799-90d5-3fa31f352a6a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"
Aug 04 18:25:15.405772 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.405740    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvlf\" (UniqueName: \"kubernetes.io/projected/7af3cc75-0e41-4799-90d5-3fa31f352a6a-kube-api-access-9zvlf\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw\" (UID: \"7af3cc75-0e41-4799-90d5-3fa31f352a6a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"
Aug 04 18:25:15.508909 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.508809    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"
Aug 04 18:25:15.649962 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:15.649928    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"]
Aug 04 18:25:15.653099 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:25:15.653075    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af3cc75_0e41_4799_90d5_3fa31f352a6a.slice/crio-90816439e32743d0b825cf94b8ab8803d1e1b4a1b4e72d54360cf59616f087df WatchSource:0}: Error finding container 90816439e32743d0b825cf94b8ab8803d1e1b4a1b4e72d54360cf59616f087df: Status 404 returned error can't find the container with id 90816439e32743d0b825cf94b8ab8803d1e1b4a1b4e72d54360cf59616f087df
Aug 04 18:25:16.646351 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:16.646316    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw" event={"ID":"7af3cc75-0e41-4799-90d5-3fa31f352a6a","Type":"ContainerStarted","Data":"b7267e64b42f4acd5f16f082d57f3d4d9d5a57feddd6994c56c2b5e090c898cf"}
Aug 04 18:25:16.646757 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:16.646367    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw" event={"ID":"7af3cc75-0e41-4799-90d5-3fa31f352a6a","Type":"ContainerStarted","Data":"90816439e32743d0b825cf94b8ab8803d1e1b4a1b4e72d54360cf59616f087df"}
Aug 04 18:25:16.646757 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:16.646454    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"
Aug 04 18:25:16.669301 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:16.669249    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw" podStartSLOduration=1.6692339 podStartE2EDuration="1.6692339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:25:16.666976919 +0000 UTC m=+1516.642808616" watchObservedRunningTime="2026-08-04 18:25:16.6692339 +0000 UTC m=+1516.645065597"
Aug 04 18:25:47.039828 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:47.039790    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:25:47.653002 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:25:47.652972    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"
Aug 04 18:26:30.973550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:30.973513    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:30.974181 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:30.973999    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"]
Aug 04 18:26:32.831576 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:32.831539    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:37.653724 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:37.653681    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw" podUID="7af3cc75-0e41-4799-90d5-3fa31f352a6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:26:45.998287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:45.998229    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw" podUID="7af3cc75-0e41-4799-90d5-3fa31f352a6a" containerName="main" containerID="cri-o://b7267e64b42f4acd5f16f082d57f3d4d9d5a57feddd6994c56c2b5e090c898cf" gracePeriod=45
Aug 04 18:26:46.251415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:46.251350    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"
Aug 04 18:26:46.328611 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:46.328578    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af3cc75-0e41-4799-90d5-3fa31f352a6a-tls-certs\") pod \"7af3cc75-0e41-4799-90d5-3fa31f352a6a\" (UID: \"7af3cc75-0e41-4799-90d5-3fa31f352a6a\") "
Aug 04 18:26:46.328794 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:46.328642    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zvlf\" (UniqueName: \"kubernetes.io/projected/7af3cc75-0e41-4799-90d5-3fa31f352a6a-kube-api-access-9zvlf\") pod \"7af3cc75-0e41-4799-90d5-3fa31f352a6a\" (UID: \"7af3cc75-0e41-4799-90d5-3fa31f352a6a\") "
Aug 04 18:26:46.330799 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:46.330769    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af3cc75-0e41-4799-90d5-3fa31f352a6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7af3cc75-0e41-4799-90d5-3fa31f352a6a" (UID: "7af3cc75-0e41-4799-90d5-3fa31f352a6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:26:46.330799 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:46.330784    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af3cc75-0e41-4799-90d5-3fa31f352a6a-kube-api-access-9zvlf" (OuterVolumeSpecName: "kube-api-access-9zvlf") pod "7af3cc75-0e41-4799-90d5-3fa31f352a6a" (UID: "7af3cc75-0e41-4799-90d5-3fa31f352a6a"). InnerVolumeSpecName "kube-api-access-9zvlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:26:46.429245 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:46.429201    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zvlf\" (UniqueName: \"kubernetes.io/projected/7af3cc75-0e41-4799-90d5-3fa31f352a6a-kube-api-access-9zvlf\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:26:46.429245 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:46.429233    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af3cc75-0e41-4799-90d5-3fa31f352a6a-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:26:47.072630 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:47.072594    2576 generic.go:356] "Generic (PLEG): container finished" podID="7af3cc75-0e41-4799-90d5-3fa31f352a6a" containerID="b7267e64b42f4acd5f16f082d57f3d4d9d5a57feddd6994c56c2b5e090c898cf" exitCode=0
Aug 04 18:26:47.073046 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:47.072641    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"
Aug 04 18:26:47.073046 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:47.072687    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw" event={"ID":"7af3cc75-0e41-4799-90d5-3fa31f352a6a","Type":"ContainerDied","Data":"b7267e64b42f4acd5f16f082d57f3d4d9d5a57feddd6994c56c2b5e090c898cf"}
Aug 04 18:26:47.073046 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:47.072743    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw" event={"ID":"7af3cc75-0e41-4799-90d5-3fa31f352a6a","Type":"ContainerDied","Data":"90816439e32743d0b825cf94b8ab8803d1e1b4a1b4e72d54360cf59616f087df"}
Aug 04 18:26:47.073046 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:47.072765    2576 scope.go:117] "RemoveContainer" containerID="b7267e64b42f4acd5f16f082d57f3d4d9d5a57feddd6994c56c2b5e090c898cf"
Aug 04 18:26:47.081168 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:47.081147    2576 scope.go:117] "RemoveContainer" containerID="b7267e64b42f4acd5f16f082d57f3d4d9d5a57feddd6994c56c2b5e090c898cf"
Aug 04 18:26:47.081438 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:26:47.081415    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7267e64b42f4acd5f16f082d57f3d4d9d5a57feddd6994c56c2b5e090c898cf\": container with ID starting with b7267e64b42f4acd5f16f082d57f3d4d9d5a57feddd6994c56c2b5e090c898cf not found: ID does not exist" containerID="b7267e64b42f4acd5f16f082d57f3d4d9d5a57feddd6994c56c2b5e090c898cf"
Aug 04 18:26:47.081503 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:47.081446    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7267e64b42f4acd5f16f082d57f3d4d9d5a57feddd6994c56c2b5e090c898cf"} err="failed to get container status \"b7267e64b42f4acd5f16f082d57f3d4d9d5a57feddd6994c56c2b5e090c898cf\": rpc error: code = NotFound desc = could not find container \"b7267e64b42f4acd5f16f082d57f3d4d9d5a57feddd6994c56c2b5e090c898cf\": container with ID starting with b7267e64b42f4acd5f16f082d57f3d4d9d5a57feddd6994c56c2b5e090c898cf not found: ID does not exist"
Aug 04 18:26:47.091870 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:47.091837    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"]
Aug 04 18:26:47.094415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:47.094378    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65hwxfw"]
Aug 04 18:26:48.582359 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:48.582325    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7af3cc75-0e41-4799-90d5-3fa31f352a6a" path="/var/lib/kubelet/pods/7af3cc75-0e41-4799-90d5-3fa31f352a6a/volumes"
Aug 04 18:26:50.679948 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:50.679909    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:50.779576 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:50.779537    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:56.695691 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:26:56.695598    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:00.863971 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:00.863932    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:01.083390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.083343    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"]
Aug 04 18:27:01.084002 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.083975    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af3cc75-0e41-4799-90d5-3fa31f352a6a" containerName="main"
Aug 04 18:27:01.084002 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.084002    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af3cc75-0e41-4799-90d5-3fa31f352a6a" containerName="main"
Aug 04 18:27:01.084169 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.084132    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af3cc75-0e41-4799-90d5-3fa31f352a6a" containerName="main"
Aug 04 18:27:01.087334 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.087311    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"
Aug 04 18:27:01.091303 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.091275    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Aug 04 18:27:01.091303 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.091292    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-vr6g7\""
Aug 04 18:27:01.091491 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.091311    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Aug 04 18:27:01.091538 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.091501    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Aug 04 18:27:01.091581 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.091569    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-9vdc4\""
Aug 04 18:27:01.094655 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.094627    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:01.096856 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.096833    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"]
Aug 04 18:27:01.152661 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.152630    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:01.163715 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.163680    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6w82\" (UniqueName: \"kubernetes.io/projected/f4a7e217-2c0f-4608-979d-0f17d7d55791-kube-api-access-j6w82\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf\" (UID: \"f4a7e217-2c0f-4608-979d-0f17d7d55791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"
Aug 04 18:27:01.163866 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.163797    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4a7e217-2c0f-4608-979d-0f17d7d55791-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf\" (UID: \"f4a7e217-2c0f-4608-979d-0f17d7d55791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"
Aug 04 18:27:01.265054 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.265019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4a7e217-2c0f-4608-979d-0f17d7d55791-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf\" (UID: \"f4a7e217-2c0f-4608-979d-0f17d7d55791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"
Aug 04 18:27:01.265054 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.265060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6w82\" (UniqueName: \"kubernetes.io/projected/f4a7e217-2c0f-4608-979d-0f17d7d55791-kube-api-access-j6w82\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf\" (UID: \"f4a7e217-2c0f-4608-979d-0f17d7d55791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"
Aug 04 18:27:01.267754 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.267727    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4a7e217-2c0f-4608-979d-0f17d7d55791-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf\" (UID: \"f4a7e217-2c0f-4608-979d-0f17d7d55791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"
Aug 04 18:27:01.273875 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.273850    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6w82\" (UniqueName: \"kubernetes.io/projected/f4a7e217-2c0f-4608-979d-0f17d7d55791-kube-api-access-j6w82\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf\" (UID: \"f4a7e217-2c0f-4608-979d-0f17d7d55791\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"
Aug 04 18:27:01.400087 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.399992    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"
Aug 04 18:27:01.546299 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:01.546270    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"]
Aug 04 18:27:01.548804 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:27:01.548777    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a7e217_2c0f_4608_979d_0f17d7d55791.slice/crio-f2679c896d5d09737d1a088c3d062e6738614d6703cdb01b6a20e8694330342a WatchSource:0}: Error finding container f2679c896d5d09737d1a088c3d062e6738614d6703cdb01b6a20e8694330342a: Status 404 returned error can't find the container with id f2679c896d5d09737d1a088c3d062e6738614d6703cdb01b6a20e8694330342a
Aug 04 18:27:02.146226 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:02.146185    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf" event={"ID":"f4a7e217-2c0f-4608-979d-0f17d7d55791","Type":"ContainerStarted","Data":"11298196ed86f52d7fc9dcc016b595fbaa707d82d3dce69befb9ec34c1379845"}
Aug 04 18:27:02.146738 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:02.146249    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf" event={"ID":"f4a7e217-2c0f-4608-979d-0f17d7d55791","Type":"ContainerStarted","Data":"f2679c896d5d09737d1a088c3d062e6738614d6703cdb01b6a20e8694330342a"}
Aug 04 18:27:02.146738 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:02.146359    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"
Aug 04 18:27:02.166269 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:02.166213    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf" podStartSLOduration=1.166193124 podStartE2EDuration="1.166193124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:27:02.164740937 +0000 UTC m=+1622.140572636" watchObservedRunningTime="2026-08-04 18:27:02.166193124 +0000 UTC m=+1622.142024823"
Aug 04 18:27:02.976197 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:02.976151    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:03.074984 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:03.074949    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:08.019608 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:08.019569    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:08.119365 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:08.119335    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:08.219398 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:08.219363    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:12.999415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:12.999377    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"]
Aug 04 18:27:13.094193 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:13.094156    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:13.374520 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:13.374478    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:14.895769 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:14.895736    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:15.707870 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:15.707835    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:18.824710 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:18.824678    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:28.019345 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.019303    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv" podUID="a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb" containerName="main" containerID="cri-o://5112177dd88afddbadfe9d78b990fc096872c82d34647755433c93e915fd9043" gracePeriod=45
Aug 04 18:27:28.266753 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.266726    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:27:28.282397 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.282313    2576 generic.go:356] "Generic (PLEG): container finished" podID="a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb" containerID="5112177dd88afddbadfe9d78b990fc096872c82d34647755433c93e915fd9043" exitCode=0
Aug 04 18:27:28.282397 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.282362    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"
Aug 04 18:27:28.282575 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.282397    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv" event={"ID":"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb","Type":"ContainerDied","Data":"5112177dd88afddbadfe9d78b990fc096872c82d34647755433c93e915fd9043"}
Aug 04 18:27:28.282575 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.282473    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv" event={"ID":"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb","Type":"ContainerDied","Data":"3383b1af82b29deab919d2572729a403dd926f274ee8c0ff89c55686539dd0f1"}
Aug 04 18:27:28.282575 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.282497    2576 scope.go:117] "RemoveContainer" containerID="5112177dd88afddbadfe9d78b990fc096872c82d34647755433c93e915fd9043"
Aug 04 18:27:28.292885 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.292859    2576 scope.go:117] "RemoveContainer" containerID="5112177dd88afddbadfe9d78b990fc096872c82d34647755433c93e915fd9043"
Aug 04 18:27:28.293178 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:27:28.293154    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5112177dd88afddbadfe9d78b990fc096872c82d34647755433c93e915fd9043\": container with ID starting with 5112177dd88afddbadfe9d78b990fc096872c82d34647755433c93e915fd9043 not found: ID does not exist" containerID="5112177dd88afddbadfe9d78b990fc096872c82d34647755433c93e915fd9043"
Aug 04 18:27:28.293281 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.293184    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5112177dd88afddbadfe9d78b990fc096872c82d34647755433c93e915fd9043"} err="failed to get container status \"5112177dd88afddbadfe9d78b990fc096872c82d34647755433c93e915fd9043\": rpc error: code = NotFound desc = could not find container \"5112177dd88afddbadfe9d78b990fc096872c82d34647755433c93e915fd9043\": container with ID starting with 5112177dd88afddbadfe9d78b990fc096872c82d34647755433c93e915fd9043 not found: ID does not exist"
Aug 04 18:27:28.315559 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.315532    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f9zb\" (UniqueName: \"kubernetes.io/projected/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-kube-api-access-5f9zb\") pod \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") "
Aug 04 18:27:28.315702 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.315571    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-dshm\") pod \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") "
Aug 04 18:27:28.315702 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.315597    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-tls-certs\") pod \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") "
Aug 04 18:27:28.315702 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.315651    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-home\") pod \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") "
Aug 04 18:27:28.315854 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.315727    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-model-cache\") pod \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") "
Aug 04 18:27:28.315854 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.315761    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-tmp-dir\") pod \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\" (UID: \"a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb\") "
Aug 04 18:27:28.316049 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.316013    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-home" (OuterVolumeSpecName: "home") pod "a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb" (UID: "a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:28.316170 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.316137    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb" (UID: "a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:28.316255 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.316231    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb" (UID: "a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:28.318095 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.318069    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb" (UID: "a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:27:28.318207 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.318093    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-dshm" (OuterVolumeSpecName: "dshm") pod "a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb" (UID: "a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:28.318207 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.318182    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-kube-api-access-5f9zb" (OuterVolumeSpecName: "kube-api-access-5f9zb") pod "a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb" (UID: "a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb"). InnerVolumeSpecName "kube-api-access-5f9zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:27:28.416494 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.416455    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f9zb\" (UniqueName: \"kubernetes.io/projected/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-kube-api-access-5f9zb\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:27:28.416494 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.416487    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:27:28.416494 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.416498    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:27:28.416494 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.416506    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:27:28.416764 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.416514    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:27:28.416764 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.416543    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:27:28.606121 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.606064    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"]
Aug 04 18:27:28.608284 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:28.608257    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-j9rnv"]
Aug 04 18:27:30.581874 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:30.581841    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb" path="/var/lib/kubelet/pods/a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb/volumes"
Aug 04 18:27:30.735501 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:30.735470    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:30.836986 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:30.836883    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:32.811985 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.811951    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"]
Aug 04 18:27:32.812549 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.812531    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb" containerName="main"
Aug 04 18:27:32.812619 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.812551    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb" containerName="main"
Aug 04 18:27:32.812673 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.812656    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a44cf1c8-52a9-4cdf-9210-7a0bf3fe97fb" containerName="main"
Aug 04 18:27:32.815833 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.815813    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.819741 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.819721    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Aug 04 18:27:32.819870 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.819724    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Aug 04 18:27:32.819870 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.819724    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-b96fh\""
Aug 04 18:27:32.819870 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.819814    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Aug 04 18:27:32.826775 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.826752    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"]
Aug 04 18:27:32.851300 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.851262    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:32.854699 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.854678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.854799 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.854707    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.854799 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.854726    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.854872 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.854820    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h6s5\" (UniqueName: \"kubernetes.io/projected/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-kube-api-access-4h6s5\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.854872 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.854861    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.854939 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.854915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-home\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.956237 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.956193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h6s5\" (UniqueName: \"kubernetes.io/projected/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-kube-api-access-4h6s5\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.956237 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.956244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.956450 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.956281    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-home\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.956450 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.956320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.956450 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.956337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.956450 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.956361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.956775 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.956741    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-home\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.956775 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.956755    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.956968 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.956797    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.958617 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.958586    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.958803 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.958788    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:32.965072 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:32.965039    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h6s5\" (UniqueName: \"kubernetes.io/projected/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-kube-api-access-4h6s5\") pod \"test-div-v2-kserve-5f6fdb6b77-t7r6t\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:33.127963 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:33.127886    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:33.153873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:33.153839    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"
Aug 04 18:27:33.278443 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:33.278411    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"]
Aug 04 18:27:33.278913 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:27:33.278884    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e29e7d_76c4_4b71_996b_24dbdcc5f0ed.slice/crio-87eb63b54ca6ab2abdb6676e87e1b40827026bf206796d064207bd03ccc11f7d WatchSource:0}: Error finding container 87eb63b54ca6ab2abdb6676e87e1b40827026bf206796d064207bd03ccc11f7d: Status 404 returned error can't find the container with id 87eb63b54ca6ab2abdb6676e87e1b40827026bf206796d064207bd03ccc11f7d
Aug 04 18:27:33.306939 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:33.306912    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t" event={"ID":"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed","Type":"ContainerStarted","Data":"87eb63b54ca6ab2abdb6676e87e1b40827026bf206796d064207bd03ccc11f7d"}
Aug 04 18:27:34.077486 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:34.077435    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:34.313054 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:34.313015    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t" event={"ID":"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed","Type":"ContainerStarted","Data":"bff4fa603b99bb7ea5fbc201695fd97bad24df1057ac108b2da04f440cc8e12d"}
Aug 04 18:27:34.332106 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:34.332012    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t" podStartSLOduration=2.331994989 podStartE2EDuration="2.331994989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:27:34.330598677 +0000 UTC m=+1654.306430388" watchObservedRunningTime="2026-08-04 18:27:34.331994989 +0000 UTC m=+1654.307826686"
Aug 04 18:27:35.128586 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:35.128549    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:43.128595 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:43.128557    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:43.141458 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:43.141433    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:43.367910 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:43.367883    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:27:50.141662 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:50.141615    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:55.785820 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:55.785780    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:55.841332 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:55.841274    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:56.548060 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:56.547889    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:56.556364 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:56.556336    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"]
Aug 04 18:27:56.611531 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:56.611488    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:58.941609 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:58.941575    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:59.077779 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:59.077740    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:59.164966 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:59.164927    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:59.296343 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:59.296305    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"]
Aug 04 18:27:59.477245 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:27:59.477215    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:03.153179 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:03.153139    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf" podUID="f4a7e217-2c0f-4608-979d-0f17d7d55791" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:28:04.008854 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:04.008816    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:11.582981 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:11.582945    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf" podUID="f4a7e217-2c0f-4608-979d-0f17d7d55791" containerName="main" containerID="cri-o://11298196ed86f52d7fc9dcc016b595fbaa707d82d3dce69befb9ec34c1379845" gracePeriod=45
Aug 04 18:28:11.830186 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:11.830161    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"
Aug 04 18:28:11.918777 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:11.918692    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4a7e217-2c0f-4608-979d-0f17d7d55791-tls-certs\") pod \"f4a7e217-2c0f-4608-979d-0f17d7d55791\" (UID: \"f4a7e217-2c0f-4608-979d-0f17d7d55791\") "
Aug 04 18:28:11.918931 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:11.918792    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6w82\" (UniqueName: \"kubernetes.io/projected/f4a7e217-2c0f-4608-979d-0f17d7d55791-kube-api-access-j6w82\") pod \"f4a7e217-2c0f-4608-979d-0f17d7d55791\" (UID: \"f4a7e217-2c0f-4608-979d-0f17d7d55791\") "
Aug 04 18:28:11.920845 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:11.920812    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a7e217-2c0f-4608-979d-0f17d7d55791-kube-api-access-j6w82" (OuterVolumeSpecName: "kube-api-access-j6w82") pod "f4a7e217-2c0f-4608-979d-0f17d7d55791" (UID: "f4a7e217-2c0f-4608-979d-0f17d7d55791"). InnerVolumeSpecName "kube-api-access-j6w82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:28:11.920952 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:11.920891    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a7e217-2c0f-4608-979d-0f17d7d55791-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4a7e217-2c0f-4608-979d-0f17d7d55791" (UID: "f4a7e217-2c0f-4608-979d-0f17d7d55791"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:28:12.019430 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:12.019394    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6w82\" (UniqueName: \"kubernetes.io/projected/f4a7e217-2c0f-4608-979d-0f17d7d55791-kube-api-access-j6w82\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:28:12.019430 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:12.019424    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4a7e217-2c0f-4608-979d-0f17d7d55791-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:28:12.501517 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:12.501483    2576 generic.go:356] "Generic (PLEG): container finished" podID="f4a7e217-2c0f-4608-979d-0f17d7d55791" containerID="11298196ed86f52d7fc9dcc016b595fbaa707d82d3dce69befb9ec34c1379845" exitCode=0
Aug 04 18:28:12.501668 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:12.501531    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"
Aug 04 18:28:12.501668 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:12.501572    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf" event={"ID":"f4a7e217-2c0f-4608-979d-0f17d7d55791","Type":"ContainerDied","Data":"11298196ed86f52d7fc9dcc016b595fbaa707d82d3dce69befb9ec34c1379845"}
Aug 04 18:28:12.501668 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:12.501630    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf" event={"ID":"f4a7e217-2c0f-4608-979d-0f17d7d55791","Type":"ContainerDied","Data":"f2679c896d5d09737d1a088c3d062e6738614d6703cdb01b6a20e8694330342a"}
Aug 04 18:28:12.501668 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:12.501661    2576 scope.go:117] "RemoveContainer" containerID="11298196ed86f52d7fc9dcc016b595fbaa707d82d3dce69befb9ec34c1379845"
Aug 04 18:28:12.510608 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:12.510588    2576 scope.go:117] "RemoveContainer" containerID="11298196ed86f52d7fc9dcc016b595fbaa707d82d3dce69befb9ec34c1379845"
Aug 04 18:28:12.510862 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:28:12.510840    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11298196ed86f52d7fc9dcc016b595fbaa707d82d3dce69befb9ec34c1379845\": container with ID starting with 11298196ed86f52d7fc9dcc016b595fbaa707d82d3dce69befb9ec34c1379845 not found: ID does not exist" containerID="11298196ed86f52d7fc9dcc016b595fbaa707d82d3dce69befb9ec34c1379845"
Aug 04 18:28:12.510901 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:12.510871    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11298196ed86f52d7fc9dcc016b595fbaa707d82d3dce69befb9ec34c1379845"} err="failed to get container status \"11298196ed86f52d7fc9dcc016b595fbaa707d82d3dce69befb9ec34c1379845\": rpc error: code = NotFound desc = could not find container \"11298196ed86f52d7fc9dcc016b595fbaa707d82d3dce69befb9ec34c1379845\": container with ID starting with 11298196ed86f52d7fc9dcc016b595fbaa707d82d3dce69befb9ec34c1379845 not found: ID does not exist"
Aug 04 18:28:12.524090 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:12.524053    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"]
Aug 04 18:28:12.527927 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:12.527902    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b98xvvf"]
Aug 04 18:28:12.582860 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:12.582829    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4a7e217-2c0f-4608-979d-0f17d7d55791" path="/var/lib/kubelet/pods/f4a7e217-2c0f-4608-979d-0f17d7d55791/volumes"
Aug 04 18:28:13.133006 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:13.132975    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:14.319694 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.319657    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t" podUID="55e29e7d-76c4-4b71-996b-24dbdcc5f0ed" containerName="main" containerID="cri-o://bff4fa603b99bb7ea5fbc201695fd97bad24df1057ac108b2da04f440cc8e12d" gracePeriod=45
Aug 04 18:28:14.357971 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.357928    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t" podUID="55e29e7d-76c4-4b71-996b-24dbdcc5f0ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Aug 04 18:28:14.517184 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.517146    2576 generic.go:356] "Generic (PLEG): container finished" podID="55e29e7d-76c4-4b71-996b-24dbdcc5f0ed" containerID="bff4fa603b99bb7ea5fbc201695fd97bad24df1057ac108b2da04f440cc8e12d" exitCode=0
Aug 04 18:28:14.517338 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.517152    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t" event={"ID":"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed","Type":"ContainerDied","Data":"bff4fa603b99bb7ea5fbc201695fd97bad24df1057ac108b2da04f440cc8e12d"}
Aug 04 18:28:14.571204 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.571137    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:28:14.641079 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.641044    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h6s5\" (UniqueName: \"kubernetes.io/projected/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-kube-api-access-4h6s5\") pod \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") "
Aug 04 18:28:14.641079 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.641081    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-tls-certs\") pod \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") "
Aug 04 18:28:14.641334 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.641138    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-model-cache\") pod \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") "
Aug 04 18:28:14.641334 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.641178    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-home\") pod \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") "
Aug 04 18:28:14.641334 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.641250    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-dshm\") pod \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") "
Aug 04 18:28:14.641334 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.641276    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-tmp-dir\") pod \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\" (UID: \"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed\") "
Aug 04 18:28:14.641536 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.641449    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "55e29e7d-76c4-4b71-996b-24dbdcc5f0ed" (UID: "55e29e7d-76c4-4b71-996b-24dbdcc5f0ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:28:14.641648 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.641628    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:28:14.641859 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.641836    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55e29e7d-76c4-4b71-996b-24dbdcc5f0ed" (UID: "55e29e7d-76c4-4b71-996b-24dbdcc5f0ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:28:14.641925 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.641902    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-home" (OuterVolumeSpecName: "home") pod "55e29e7d-76c4-4b71-996b-24dbdcc5f0ed" (UID: "55e29e7d-76c4-4b71-996b-24dbdcc5f0ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:28:14.643505 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.643475    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-dshm" (OuterVolumeSpecName: "dshm") pod "55e29e7d-76c4-4b71-996b-24dbdcc5f0ed" (UID: "55e29e7d-76c4-4b71-996b-24dbdcc5f0ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:28:14.643630 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.643498    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-kube-api-access-4h6s5" (OuterVolumeSpecName: "kube-api-access-4h6s5") pod "55e29e7d-76c4-4b71-996b-24dbdcc5f0ed" (UID: "55e29e7d-76c4-4b71-996b-24dbdcc5f0ed"). InnerVolumeSpecName "kube-api-access-4h6s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:28:14.643630 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.643533    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55e29e7d-76c4-4b71-996b-24dbdcc5f0ed" (UID: "55e29e7d-76c4-4b71-996b-24dbdcc5f0ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:28:14.742717 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.742679    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h6s5\" (UniqueName: \"kubernetes.io/projected/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-kube-api-access-4h6s5\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:28:14.742717 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.742710    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:28:14.742717 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.742720    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:28:14.742717 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.742728    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:28:14.742981 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:14.742736    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:28:15.523302 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:15.523266    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t" event={"ID":"55e29e7d-76c4-4b71-996b-24dbdcc5f0ed","Type":"ContainerDied","Data":"87eb63b54ca6ab2abdb6676e87e1b40827026bf206796d064207bd03ccc11f7d"}
Aug 04 18:28:15.523302 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:15.523290    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"
Aug 04 18:28:15.523832 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:15.523337    2576 scope.go:117] "RemoveContainer" containerID="bff4fa603b99bb7ea5fbc201695fd97bad24df1057ac108b2da04f440cc8e12d"
Aug 04 18:28:15.544395 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:15.544360    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"]
Aug 04 18:28:15.548030 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:15.548005    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-t7r6t"]
Aug 04 18:28:16.444432 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:16.444392    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:16.544048 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:16.544009    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:16.581845 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:16.581810    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55e29e7d-76c4-4b71-996b-24dbdcc5f0ed" path="/var/lib/kubelet/pods/55e29e7d-76c4-4b71-996b-24dbdcc5f0ed/volumes"
Aug 04 18:28:22.297852 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.297774    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"]
Aug 04 18:28:22.298222 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.298160    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55e29e7d-76c4-4b71-996b-24dbdcc5f0ed" containerName="main"
Aug 04 18:28:22.298222 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.298174    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="55e29e7d-76c4-4b71-996b-24dbdcc5f0ed" containerName="main"
Aug 04 18:28:22.298222 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.298190    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a7e217-2c0f-4608-979d-0f17d7d55791" containerName="main"
Aug 04 18:28:22.298222 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.298195    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a7e217-2c0f-4608-979d-0f17d7d55791" containerName="main"
Aug 04 18:28:22.298367 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.298263    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a7e217-2c0f-4608-979d-0f17d7d55791" containerName="main"
Aug 04 18:28:22.298367 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.298277    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="55e29e7d-76c4-4b71-996b-24dbdcc5f0ed" containerName="main"
Aug 04 18:28:22.302446 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.302425    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.306013 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.305975    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Aug 04 18:28:22.306013 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.305997    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-cxsbt\""
Aug 04 18:28:22.306245 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.305975    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Aug 04 18:28:22.306245 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.306045    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Aug 04 18:28:22.323731 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.323702    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"]
Aug 04 18:28:22.347444 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.347411    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:22.409400 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.409365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.409576 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.409446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90044b3-ecd7-4a2e-9605-900821359fbd-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.409576 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.409517    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.409576 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.409551    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdsvb\" (UniqueName: \"kubernetes.io/projected/d90044b3-ecd7-4a2e-9605-900821359fbd-kube-api-access-jdsvb\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.409576 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.409574    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-home\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.409741 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.409605    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.510215 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.510183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.510414 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.510227    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90044b3-ecd7-4a2e-9605-900821359fbd-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.510414 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.510266    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.510414 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.510296    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdsvb\" (UniqueName: \"kubernetes.io/projected/d90044b3-ecd7-4a2e-9605-900821359fbd-kube-api-access-jdsvb\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.510414 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.510317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-home\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.510414 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.510389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.510694 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.510649    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.510762 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.510693    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.510762 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.510738    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-home\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.512626 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.512606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.519898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.513219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90044b3-ecd7-4a2e-9605-900821359fbd-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.519898 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.519397    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdsvb\" (UniqueName: \"kubernetes.io/projected/d90044b3-ecd7-4a2e-9605-900821359fbd-kube-api-access-jdsvb\") pod \"test-div-v2-kserve-5f6fdb6b77-6qh9c\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.613292 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.613185    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:22.751539 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.751513    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"]
Aug 04 18:28:22.755190 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:28:22.754771    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90044b3_ecd7_4a2e_9605_900821359fbd.slice/crio-eb4c8de696a2e7d8afa9c92b366fe07db3dc89b79be84f0275b8df504b81ca05 WatchSource:0}: Error finding container eb4c8de696a2e7d8afa9c92b366fe07db3dc89b79be84f0275b8df504b81ca05: Status 404 returned error can't find the container with id eb4c8de696a2e7d8afa9c92b366fe07db3dc89b79be84f0275b8df504b81ca05
Aug 04 18:28:22.920740 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:22.920593    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:23.562764 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:23.562728    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c" event={"ID":"d90044b3-ecd7-4a2e-9605-900821359fbd","Type":"ContainerStarted","Data":"569f296a097fc4cee2e72931c690044b46ce5cfb79e3736cf682d21f3add8e7d"}
Aug 04 18:28:23.563163 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:23.562780    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c" event={"ID":"d90044b3-ecd7-4a2e-9605-900821359fbd","Type":"ContainerStarted","Data":"eb4c8de696a2e7d8afa9c92b366fe07db3dc89b79be84f0275b8df504b81ca05"}
Aug 04 18:28:23.582194 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:23.582137    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c" podStartSLOduration=1.582105476 podStartE2EDuration="1.582105476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:28:23.579768026 +0000 UTC m=+1703.555599722" watchObservedRunningTime="2026-08-04 18:28:23.582105476 +0000 UTC m=+1703.557937174"
Aug 04 18:28:23.613838 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:23.613801    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:28.474499 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:28.474460    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:28.530184 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:28.530151    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:32.613825 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:32.613791    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:32.626269 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:32.626238    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:33.619853 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:33.619823    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:28:40.983590 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:40.983546    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:42.057156 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.057085    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:42.317149 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.317058    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"]
Aug 04 18:28:42.320731 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.320707    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"
Aug 04 18:28:42.323235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.323157    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:42.323235 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.323224    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Aug 04 18:28:42.324178 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.324159    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-gdf9r\""
Aug 04 18:28:42.324314 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.324286    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Aug 04 18:28:42.324383 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.324321    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-9qrb5\""
Aug 04 18:28:42.324426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.324322    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Aug 04 18:28:42.328056 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.328035    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"]
Aug 04 18:28:42.400556 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.400511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffb6\" (UniqueName: \"kubernetes.io/projected/67002839-223a-45ce-a98e-be4511b277f3-kube-api-access-6ffb6\") pod \"tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9\" (UID: \"67002839-223a-45ce-a98e-be4511b277f3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"
Aug 04 18:28:42.400717 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.400570    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67002839-223a-45ce-a98e-be4511b277f3-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9\" (UID: \"67002839-223a-45ce-a98e-be4511b277f3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"
Aug 04 18:28:42.420661 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.420630    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:42.501812 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.501775    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffb6\" (UniqueName: \"kubernetes.io/projected/67002839-223a-45ce-a98e-be4511b277f3-kube-api-access-6ffb6\") pod \"tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9\" (UID: \"67002839-223a-45ce-a98e-be4511b277f3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"
Aug 04 18:28:42.501998 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.501843    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67002839-223a-45ce-a98e-be4511b277f3-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9\" (UID: \"67002839-223a-45ce-a98e-be4511b277f3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"
Aug 04 18:28:42.504396 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.504374    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67002839-223a-45ce-a98e-be4511b277f3-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9\" (UID: \"67002839-223a-45ce-a98e-be4511b277f3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"
Aug 04 18:28:42.510451 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.510425    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffb6\" (UniqueName: \"kubernetes.io/projected/67002839-223a-45ce-a98e-be4511b277f3-kube-api-access-6ffb6\") pod \"tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9\" (UID: \"67002839-223a-45ce-a98e-be4511b277f3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"
Aug 04 18:28:42.633599 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.633506    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"
Aug 04 18:28:42.774847 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:42.774823    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"]
Aug 04 18:28:42.777070 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:28:42.777041    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67002839_223a_45ce_a98e_be4511b277f3.slice/crio-2066883feb6f7b450d0797e8b2c67e5c2f80beba433548cbb7fd7a5f232e75df WatchSource:0}: Error finding container 2066883feb6f7b450d0797e8b2c67e5c2f80beba433548cbb7fd7a5f232e75df: Status 404 returned error can't find the container with id 2066883feb6f7b450d0797e8b2c67e5c2f80beba433548cbb7fd7a5f232e75df
Aug 04 18:28:43.659821 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:43.659787    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9" event={"ID":"67002839-223a-45ce-a98e-be4511b277f3","Type":"ContainerStarted","Data":"bea381e5bc551b6b2e857d2b84e332f91165152fc2c1e92ae9dde248d1bef398"}
Aug 04 18:28:43.660303 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:43.659841    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9" event={"ID":"67002839-223a-45ce-a98e-be4511b277f3","Type":"ContainerStarted","Data":"2066883feb6f7b450d0797e8b2c67e5c2f80beba433548cbb7fd7a5f232e75df"}
Aug 04 18:28:43.660303 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:43.659872    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"
Aug 04 18:28:43.679063 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:43.679005    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9" podStartSLOduration=1.678983883 podStartE2EDuration="1.678983883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:28:43.677896354 +0000 UTC m=+1723.653728052" watchObservedRunningTime="2026-08-04 18:28:43.678983883 +0000 UTC m=+1723.654815580"
Aug 04 18:28:47.310380 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:47.310342    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:47.410663 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:28:47.410628    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:14.666588 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:14.666555    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"
Aug 04 18:29:17.744360 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:17.744324    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:18.065502 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:18.065470    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:21.327221 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:21.327179    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:21.349834 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:21.349796    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"]
Aug 04 18:29:21.394625 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:21.394593    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:23.447873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:23.447833    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:23.552902 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:23.552858    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:23.646999 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:23.646968    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:23.919458 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:23.919420    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:24.020316 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:24.020280    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:24.142514 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:24.142474    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:24.613606 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:24.613570    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:24.666408 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:24.666370    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9" podUID="67002839-223a-45ce-a98e-be4511b277f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:29:28.426962 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:28.426924    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"]
Aug 04 18:29:28.521784 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:28.521740    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:29.650590 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:29.650552    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:34.060563 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:34.060520    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:34.665979 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:34.665936    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9" podUID="67002839-223a-45ce-a98e-be4511b277f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:29:35.098812 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:35.098775    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:35.635269 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:35.635234    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:35.720596 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:35.720556    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:36.373915 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.373875    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9" podUID="67002839-223a-45ce-a98e-be4511b277f3" containerName="main" containerID="cri-o://bea381e5bc551b6b2e857d2b84e332f91165152fc2c1e92ae9dde248d1bef398" gracePeriod=45
Aug 04 18:29:36.619415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.619391    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"
Aug 04 18:29:36.685764 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.685675    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ffb6\" (UniqueName: \"kubernetes.io/projected/67002839-223a-45ce-a98e-be4511b277f3-kube-api-access-6ffb6\") pod \"67002839-223a-45ce-a98e-be4511b277f3\" (UID: \"67002839-223a-45ce-a98e-be4511b277f3\") "
Aug 04 18:29:36.685937 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.685769    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67002839-223a-45ce-a98e-be4511b277f3-tls-certs\") pod \"67002839-223a-45ce-a98e-be4511b277f3\" (UID: \"67002839-223a-45ce-a98e-be4511b277f3\") "
Aug 04 18:29:36.687926 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.687887    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67002839-223a-45ce-a98e-be4511b277f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67002839-223a-45ce-a98e-be4511b277f3" (UID: "67002839-223a-45ce-a98e-be4511b277f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:29:36.688041 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.687932    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67002839-223a-45ce-a98e-be4511b277f3-kube-api-access-6ffb6" (OuterVolumeSpecName: "kube-api-access-6ffb6") pod "67002839-223a-45ce-a98e-be4511b277f3" (UID: "67002839-223a-45ce-a98e-be4511b277f3"). InnerVolumeSpecName "kube-api-access-6ffb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:29:36.786499 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.786467    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ffb6\" (UniqueName: \"kubernetes.io/projected/67002839-223a-45ce-a98e-be4511b277f3-kube-api-access-6ffb6\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:29:36.786499 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.786497    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67002839-223a-45ce-a98e-be4511b277f3-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:29:36.912416 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.912385    2576 generic.go:356] "Generic (PLEG): container finished" podID="67002839-223a-45ce-a98e-be4511b277f3" containerID="bea381e5bc551b6b2e857d2b84e332f91165152fc2c1e92ae9dde248d1bef398" exitCode=0
Aug 04 18:29:36.912579 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.912429    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"
Aug 04 18:29:36.912579 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.912470    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9" event={"ID":"67002839-223a-45ce-a98e-be4511b277f3","Type":"ContainerDied","Data":"bea381e5bc551b6b2e857d2b84e332f91165152fc2c1e92ae9dde248d1bef398"}
Aug 04 18:29:36.912579 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.912524    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9" event={"ID":"67002839-223a-45ce-a98e-be4511b277f3","Type":"ContainerDied","Data":"2066883feb6f7b450d0797e8b2c67e5c2f80beba433548cbb7fd7a5f232e75df"}
Aug 04 18:29:36.912579 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.912550    2576 scope.go:117] "RemoveContainer" containerID="bea381e5bc551b6b2e857d2b84e332f91165152fc2c1e92ae9dde248d1bef398"
Aug 04 18:29:36.921640 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.921620    2576 scope.go:117] "RemoveContainer" containerID="bea381e5bc551b6b2e857d2b84e332f91165152fc2c1e92ae9dde248d1bef398"
Aug 04 18:29:36.921879 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:29:36.921862    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea381e5bc551b6b2e857d2b84e332f91165152fc2c1e92ae9dde248d1bef398\": container with ID starting with bea381e5bc551b6b2e857d2b84e332f91165152fc2c1e92ae9dde248d1bef398 not found: ID does not exist" containerID="bea381e5bc551b6b2e857d2b84e332f91165152fc2c1e92ae9dde248d1bef398"
Aug 04 18:29:36.921924 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.921885    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea381e5bc551b6b2e857d2b84e332f91165152fc2c1e92ae9dde248d1bef398"} err="failed to get container status \"bea381e5bc551b6b2e857d2b84e332f91165152fc2c1e92ae9dde248d1bef398\": rpc error: code = NotFound desc = could not find container \"bea381e5bc551b6b2e857d2b84e332f91165152fc2c1e92ae9dde248d1bef398\": container with ID starting with bea381e5bc551b6b2e857d2b84e332f91165152fc2c1e92ae9dde248d1bef398 not found: ID does not exist"
Aug 04 18:29:36.937510 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.937458    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"]
Aug 04 18:29:36.941890 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:36.941872    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477s6pw9"]
Aug 04 18:29:38.582271 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:38.582236    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67002839-223a-45ce-a98e-be4511b277f3" path="/var/lib/kubelet/pods/67002839-223a-45ce-a98e-be4511b277f3/volumes"
Aug 04 18:29:40.003830 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:40.003796    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:40.101182 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:40.101134    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:40.201464 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:40.201423    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:43.446339 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.446291    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c" podUID="d90044b3-ecd7-4a2e-9605-900821359fbd" containerName="main" containerID="cri-o://569f296a097fc4cee2e72931c690044b46ce5cfb79e3736cf682d21f3add8e7d" gracePeriod=45
Aug 04 18:29:43.610242 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.610194    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c" podUID="d90044b3-ecd7-4a2e-9605-900821359fbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Aug 04 18:29:43.703565 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.703502    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:29:43.741783 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.741743    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdsvb\" (UniqueName: \"kubernetes.io/projected/d90044b3-ecd7-4a2e-9605-900821359fbd-kube-api-access-jdsvb\") pod \"d90044b3-ecd7-4a2e-9605-900821359fbd\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") "
Aug 04 18:29:43.741957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.741804    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-home\") pod \"d90044b3-ecd7-4a2e-9605-900821359fbd\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") "
Aug 04 18:29:43.741957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.741869    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90044b3-ecd7-4a2e-9605-900821359fbd-tls-certs\") pod \"d90044b3-ecd7-4a2e-9605-900821359fbd\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") "
Aug 04 18:29:43.741957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.741898    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-model-cache\") pod \"d90044b3-ecd7-4a2e-9605-900821359fbd\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") "
Aug 04 18:29:43.741957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.741923    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-dshm\") pod \"d90044b3-ecd7-4a2e-9605-900821359fbd\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") "
Aug 04 18:29:43.741957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.741955    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-tmp-dir\") pod \"d90044b3-ecd7-4a2e-9605-900821359fbd\" (UID: \"d90044b3-ecd7-4a2e-9605-900821359fbd\") "
Aug 04 18:29:43.742317 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.742140    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-home" (OuterVolumeSpecName: "home") pod "d90044b3-ecd7-4a2e-9605-900821359fbd" (UID: "d90044b3-ecd7-4a2e-9605-900821359fbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:43.742317 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.742201    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-model-cache" (OuterVolumeSpecName: "model-cache") pod "d90044b3-ecd7-4a2e-9605-900821359fbd" (UID: "d90044b3-ecd7-4a2e-9605-900821359fbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:43.742420 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.742331    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:29:43.742420 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.742333    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d90044b3-ecd7-4a2e-9605-900821359fbd" (UID: "d90044b3-ecd7-4a2e-9605-900821359fbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:43.742420 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.742350    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:29:43.744361 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.744331    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90044b3-ecd7-4a2e-9605-900821359fbd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d90044b3-ecd7-4a2e-9605-900821359fbd" (UID: "d90044b3-ecd7-4a2e-9605-900821359fbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:29:43.744361 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.744349    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-dshm" (OuterVolumeSpecName: "dshm") pod "d90044b3-ecd7-4a2e-9605-900821359fbd" (UID: "d90044b3-ecd7-4a2e-9605-900821359fbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:43.744545 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.744507    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90044b3-ecd7-4a2e-9605-900821359fbd-kube-api-access-jdsvb" (OuterVolumeSpecName: "kube-api-access-jdsvb") pod "d90044b3-ecd7-4a2e-9605-900821359fbd" (UID: "d90044b3-ecd7-4a2e-9605-900821359fbd"). InnerVolumeSpecName "kube-api-access-jdsvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:29:43.843678 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.843638    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90044b3-ecd7-4a2e-9605-900821359fbd-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:29:43.843678 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.843668    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:29:43.843678 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.843679    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90044b3-ecd7-4a2e-9605-900821359fbd-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:29:43.843919 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.843691    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdsvb\" (UniqueName: \"kubernetes.io/projected/d90044b3-ecd7-4a2e-9605-900821359fbd-kube-api-access-jdsvb\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:29:43.947873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.947839    2576 generic.go:356] "Generic (PLEG): container finished" podID="d90044b3-ecd7-4a2e-9605-900821359fbd" containerID="569f296a097fc4cee2e72931c690044b46ce5cfb79e3736cf682d21f3add8e7d" exitCode=0
Aug 04 18:29:43.948047 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.947883    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"
Aug 04 18:29:43.948047 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.947894    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c" event={"ID":"d90044b3-ecd7-4a2e-9605-900821359fbd","Type":"ContainerDied","Data":"569f296a097fc4cee2e72931c690044b46ce5cfb79e3736cf682d21f3add8e7d"}
Aug 04 18:29:43.948047 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.947953    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c" event={"ID":"d90044b3-ecd7-4a2e-9605-900821359fbd","Type":"ContainerDied","Data":"eb4c8de696a2e7d8afa9c92b366fe07db3dc89b79be84f0275b8df504b81ca05"}
Aug 04 18:29:43.948047 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.947979    2576 scope.go:117] "RemoveContainer" containerID="569f296a097fc4cee2e72931c690044b46ce5cfb79e3736cf682d21f3add8e7d"
Aug 04 18:29:43.956525 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.956508    2576 scope.go:117] "RemoveContainer" containerID="569f296a097fc4cee2e72931c690044b46ce5cfb79e3736cf682d21f3add8e7d"
Aug 04 18:29:43.956809 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:29:43.956790    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"569f296a097fc4cee2e72931c690044b46ce5cfb79e3736cf682d21f3add8e7d\": container with ID starting with 569f296a097fc4cee2e72931c690044b46ce5cfb79e3736cf682d21f3add8e7d not found: ID does not exist" containerID="569f296a097fc4cee2e72931c690044b46ce5cfb79e3736cf682d21f3add8e7d"
Aug 04 18:29:43.956874 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.956822    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"569f296a097fc4cee2e72931c690044b46ce5cfb79e3736cf682d21f3add8e7d"} err="failed to get container status \"569f296a097fc4cee2e72931c690044b46ce5cfb79e3736cf682d21f3add8e7d\": rpc error: code = NotFound desc = could not find container \"569f296a097fc4cee2e72931c690044b46ce5cfb79e3736cf682d21f3add8e7d\": container with ID starting with 569f296a097fc4cee2e72931c690044b46ce5cfb79e3736cf682d21f3add8e7d not found: ID does not exist"
Aug 04 18:29:43.968100 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.968071    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"]
Aug 04 18:29:43.972061 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:43.972037    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-6qh9c"]
Aug 04 18:29:44.582172 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:44.582129    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d90044b3-ecd7-4a2e-9605-900821359fbd" path="/var/lib/kubelet/pods/d90044b3-ecd7-4a2e-9605-900821359fbd/volumes"
Aug 04 18:29:45.465930 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:45.465891    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:45.565946 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:45.565911    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:46.028908 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:46.028875    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:51.578057 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:51.578019    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:51.678134 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:51.678078    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:53.399873 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.399789    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"]
Aug 04 18:29:53.400279 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.400213    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67002839-223a-45ce-a98e-be4511b277f3" containerName="main"
Aug 04 18:29:53.400279 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.400227    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="67002839-223a-45ce-a98e-be4511b277f3" containerName="main"
Aug 04 18:29:53.400279 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.400238    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90044b3-ecd7-4a2e-9605-900821359fbd" containerName="main"
Aug 04 18:29:53.400279 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.400244    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90044b3-ecd7-4a2e-9605-900821359fbd" containerName="main"
Aug 04 18:29:53.400409 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.400323    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="67002839-223a-45ce-a98e-be4511b277f3" containerName="main"
Aug 04 18:29:53.400409 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.400333    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90044b3-ecd7-4a2e-9605-900821359fbd" containerName="main"
Aug 04 18:29:53.404911 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.404891    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.407762 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.407744    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Aug 04 18:29:53.408817 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.408797    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Aug 04 18:29:53.408918 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.408820    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Aug 04 18:29:53.409130 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.409101    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-59ldg\""
Aug 04 18:29:53.412981 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.412956    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"]
Aug 04 18:29:53.493699 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.493661    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:53.531313 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.531277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxq8z\" (UniqueName: \"kubernetes.io/projected/1d4e3e36-387e-40d8-b308-e649454b5c6c-kube-api-access-gxq8z\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.531499 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.531345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-model-cache\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.531499 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.531412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-home\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.531499 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.531454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-dshm\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.531499 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.531481    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d4e3e36-387e-40d8-b308-e649454b5c6c-tls-certs\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.531674 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.531504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.632777 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.632736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.632962 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.632789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxq8z\" (UniqueName: \"kubernetes.io/projected/1d4e3e36-387e-40d8-b308-e649454b5c6c-kube-api-access-gxq8z\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.632962 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.632844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-model-cache\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.632962 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.632901    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-home\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.632962 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.632947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-dshm\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.633250 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.632973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d4e3e36-387e-40d8-b308-e649454b5c6c-tls-certs\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.633250 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.633224    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.633250 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.633235    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-model-cache\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.633415 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.633294    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-home\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.635256 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.635235    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-dshm\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.635500 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.635483    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d4e3e36-387e-40d8-b308-e649454b5c6c-tls-certs\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.640842 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.640819    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxq8z\" (UniqueName: \"kubernetes.io/projected/1d4e3e36-387e-40d8-b308-e649454b5c6c-kube-api-access-gxq8z\") pod \"stop-v2-kserve-7f68747c45-765vv\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.716047 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.715955    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:29:53.850240 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.850207    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"]
Aug 04 18:29:53.854345 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:29:53.854311    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4e3e36_387e_40d8_b308_e649454b5c6c.slice/crio-a094129faea448e6efda251d1881f6f6f775b50ed1d7e809a3eabd4e82d0dfc3 WatchSource:0}: Error finding container a094129faea448e6efda251d1881f6f6f775b50ed1d7e809a3eabd4e82d0dfc3: Status 404 returned error can't find the container with id a094129faea448e6efda251d1881f6f6f775b50ed1d7e809a3eabd4e82d0dfc3
Aug 04 18:29:53.997575 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.997477    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv" event={"ID":"1d4e3e36-387e-40d8-b308-e649454b5c6c","Type":"ContainerStarted","Data":"9235b3227e8ad2dffbe3248dbf1e9ba4a4ddb5fe5398732e0cbd8cedffb4af62"}
Aug 04 18:29:53.997575 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:53.997542    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv" event={"ID":"1d4e3e36-387e-40d8-b308-e649454b5c6c","Type":"ContainerStarted","Data":"a094129faea448e6efda251d1881f6f6f775b50ed1d7e809a3eabd4e82d0dfc3"}
Aug 04 18:29:54.016082 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:54.016032    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv" podStartSLOduration=1.016016681 podStartE2EDuration="1.016016681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:29:54.015288962 +0000 UTC m=+1793.991120661" watchObservedRunningTime="2026-08-04 18:29:54.016016681 +0000 UTC m=+1793.991848380"
Aug 04 18:29:54.716900 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:29:54.716867    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:30:03.716248 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:03.716215    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:30:03.728927 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:03.728901    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:30:04.062489 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:04.062458    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:30:05.747072 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:05.747021    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:11.316106 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:11.316067    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:11.372984 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:11.372956    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:19.259340 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.259292    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:19.526243 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.526161    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"]
Aug 04 18:30:19.529987 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.529961    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"
Aug 04 18:30:19.534005 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.533978    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Aug 04 18:30:19.534186 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.534044    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-gvzg6\""
Aug 04 18:30:19.534186 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.534095    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-5bxcp\""
Aug 04 18:30:19.534186 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.533990    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Aug 04 18:30:19.534186 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.534166    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Aug 04 18:30:19.540052 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.540030    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"]
Aug 04 18:30:19.571577 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.571542    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:19.667759 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.667728    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdc0836-a77f-4eaf-a5d1-146870a1d356-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5\" (UID: \"1cdc0836-a77f-4eaf-a5d1-146870a1d356\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"
Aug 04 18:30:19.667910 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.667792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8b4w\" (UniqueName: \"kubernetes.io/projected/1cdc0836-a77f-4eaf-a5d1-146870a1d356-kube-api-access-t8b4w\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5\" (UID: \"1cdc0836-a77f-4eaf-a5d1-146870a1d356\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"
Aug 04 18:30:19.769332 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.769298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdc0836-a77f-4eaf-a5d1-146870a1d356-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5\" (UID: \"1cdc0836-a77f-4eaf-a5d1-146870a1d356\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"
Aug 04 18:30:19.769490 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.769379    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8b4w\" (UniqueName: \"kubernetes.io/projected/1cdc0836-a77f-4eaf-a5d1-146870a1d356-kube-api-access-t8b4w\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5\" (UID: \"1cdc0836-a77f-4eaf-a5d1-146870a1d356\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"
Aug 04 18:30:19.771642 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.771621    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdc0836-a77f-4eaf-a5d1-146870a1d356-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5\" (UID: \"1cdc0836-a77f-4eaf-a5d1-146870a1d356\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"
Aug 04 18:30:19.778052 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.777987    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8b4w\" (UniqueName: \"kubernetes.io/projected/1cdc0836-a77f-4eaf-a5d1-146870a1d356-kube-api-access-t8b4w\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5\" (UID: \"1cdc0836-a77f-4eaf-a5d1-146870a1d356\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"
Aug 04 18:30:19.843774 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.843742    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"
Aug 04 18:30:19.974804 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:19.974776    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"]
Aug 04 18:30:19.977123 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:30:19.977076    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdc0836_a77f_4eaf_a5d1_146870a1d356.slice/crio-59aeeb509d303a257ca31fd262884e27f6f5b75eca63afddd909dfd3650ff879 WatchSource:0}: Error finding container 59aeeb509d303a257ca31fd262884e27f6f5b75eca63afddd909dfd3650ff879: Status 404 returned error can't find the container with id 59aeeb509d303a257ca31fd262884e27f6f5b75eca63afddd909dfd3650ff879
Aug 04 18:30:20.131814 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:20.131775    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5" event={"ID":"1cdc0836-a77f-4eaf-a5d1-146870a1d356","Type":"ContainerStarted","Data":"79e640eaff021e6e57f3f66afcb23db71fa0d0a482967f080a4038a52817994e"}
Aug 04 18:30:20.132006 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:20.131839    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5" event={"ID":"1cdc0836-a77f-4eaf-a5d1-146870a1d356","Type":"ContainerStarted","Data":"59aeeb509d303a257ca31fd262884e27f6f5b75eca63afddd909dfd3650ff879"}
Aug 04 18:30:20.132006 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:20.131869    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"
Aug 04 18:30:20.150316 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:20.150246    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5" podStartSLOduration=1.150226742 podStartE2EDuration="1.150226742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:30:20.149704534 +0000 UTC m=+1820.125536271" watchObservedRunningTime="2026-08-04 18:30:20.150226742 +0000 UTC m=+1820.126058443"
Aug 04 18:30:27.511386 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:27.511343    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:34.975065 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:34.975030    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"]
Aug 04 18:30:35.008784 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:35.008750    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:49.995263 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:49.995212    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv" podUID="1d4e3e36-387e-40d8-b308-e649454b5c6c" containerName="main" containerID="cri-o://9235b3227e8ad2dffbe3248dbf1e9ba4a4ddb5fe5398732e0cbd8cedffb4af62" gracePeriod=45
Aug 04 18:30:50.051550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.051500    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv" podUID="1d4e3e36-387e-40d8-b308-e649454b5c6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Aug 04 18:30:50.269958 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.269929    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:30:50.284550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.284505    2576 generic.go:356] "Generic (PLEG): container finished" podID="1d4e3e36-387e-40d8-b308-e649454b5c6c" containerID="9235b3227e8ad2dffbe3248dbf1e9ba4a4ddb5fe5398732e0cbd8cedffb4af62" exitCode=0
Aug 04 18:30:50.284740 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.284560    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv" event={"ID":"1d4e3e36-387e-40d8-b308-e649454b5c6c","Type":"ContainerDied","Data":"9235b3227e8ad2dffbe3248dbf1e9ba4a4ddb5fe5398732e0cbd8cedffb4af62"}
Aug 04 18:30:50.284740 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.284611    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv" event={"ID":"1d4e3e36-387e-40d8-b308-e649454b5c6c","Type":"ContainerDied","Data":"a094129faea448e6efda251d1881f6f6f775b50ed1d7e809a3eabd4e82d0dfc3"}
Aug 04 18:30:50.284740 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.284641    2576 scope.go:117] "RemoveContainer" containerID="9235b3227e8ad2dffbe3248dbf1e9ba4a4ddb5fe5398732e0cbd8cedffb4af62"
Aug 04 18:30:50.284740 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.284679    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"
Aug 04 18:30:50.296371 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.296344    2576 scope.go:117] "RemoveContainer" containerID="9235b3227e8ad2dffbe3248dbf1e9ba4a4ddb5fe5398732e0cbd8cedffb4af62"
Aug 04 18:30:50.296725 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:30:50.296697    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9235b3227e8ad2dffbe3248dbf1e9ba4a4ddb5fe5398732e0cbd8cedffb4af62\": container with ID starting with 9235b3227e8ad2dffbe3248dbf1e9ba4a4ddb5fe5398732e0cbd8cedffb4af62 not found: ID does not exist" containerID="9235b3227e8ad2dffbe3248dbf1e9ba4a4ddb5fe5398732e0cbd8cedffb4af62"
Aug 04 18:30:50.296853 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.296727    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9235b3227e8ad2dffbe3248dbf1e9ba4a4ddb5fe5398732e0cbd8cedffb4af62"} err="failed to get container status \"9235b3227e8ad2dffbe3248dbf1e9ba4a4ddb5fe5398732e0cbd8cedffb4af62\": rpc error: code = NotFound desc = could not find container \"9235b3227e8ad2dffbe3248dbf1e9ba4a4ddb5fe5398732e0cbd8cedffb4af62\": container with ID starting with 9235b3227e8ad2dffbe3248dbf1e9ba4a4ddb5fe5398732e0cbd8cedffb4af62 not found: ID does not exist"
Aug 04 18:30:50.340364 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.340335    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d4e3e36-387e-40d8-b308-e649454b5c6c-tls-certs\") pod \"1d4e3e36-387e-40d8-b308-e649454b5c6c\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") "
Aug 04 18:30:50.340547 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.340395    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-dshm\") pod \"1d4e3e36-387e-40d8-b308-e649454b5c6c\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") "
Aug 04 18:30:50.340547 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.340438    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxq8z\" (UniqueName: \"kubernetes.io/projected/1d4e3e36-387e-40d8-b308-e649454b5c6c-kube-api-access-gxq8z\") pod \"1d4e3e36-387e-40d8-b308-e649454b5c6c\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") "
Aug 04 18:30:50.340547 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.340469    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-tmp-dir\") pod \"1d4e3e36-387e-40d8-b308-e649454b5c6c\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") "
Aug 04 18:30:50.340547 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.340510    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-home\") pod \"1d4e3e36-387e-40d8-b308-e649454b5c6c\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") "
Aug 04 18:30:50.340765 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.340574    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-model-cache\") pod \"1d4e3e36-387e-40d8-b308-e649454b5c6c\" (UID: \"1d4e3e36-387e-40d8-b308-e649454b5c6c\") "
Aug 04 18:30:50.340817 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.340775    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-home" (OuterVolumeSpecName: "home") pod "1d4e3e36-387e-40d8-b308-e649454b5c6c" (UID: "1d4e3e36-387e-40d8-b308-e649454b5c6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:30:50.340906 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.340885    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:30:50.341023 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.340999    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d4e3e36-387e-40d8-b308-e649454b5c6c" (UID: "1d4e3e36-387e-40d8-b308-e649454b5c6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:30:50.341095 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.341072    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d4e3e36-387e-40d8-b308-e649454b5c6c" (UID: "1d4e3e36-387e-40d8-b308-e649454b5c6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:30:50.342794 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.342765    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4e3e36-387e-40d8-b308-e649454b5c6c-kube-api-access-gxq8z" (OuterVolumeSpecName: "kube-api-access-gxq8z") pod "1d4e3e36-387e-40d8-b308-e649454b5c6c" (UID: "1d4e3e36-387e-40d8-b308-e649454b5c6c"). InnerVolumeSpecName "kube-api-access-gxq8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:30:50.342909 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.342854    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d4e3e36-387e-40d8-b308-e649454b5c6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d4e3e36-387e-40d8-b308-e649454b5c6c" (UID: "1d4e3e36-387e-40d8-b308-e649454b5c6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:30:50.343067 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.343050    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-dshm" (OuterVolumeSpecName: "dshm") pod "1d4e3e36-387e-40d8-b308-e649454b5c6c" (UID: "1d4e3e36-387e-40d8-b308-e649454b5c6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:30:50.442005 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.441974    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:30:50.442005 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.442000    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d4e3e36-387e-40d8-b308-e649454b5c6c-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:30:50.442005 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.442009    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:30:50.442250 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.442017    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxq8z\" (UniqueName: \"kubernetes.io/projected/1d4e3e36-387e-40d8-b308-e649454b5c6c-kube-api-access-gxq8z\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:30:50.442250 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.442025    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4e3e36-387e-40d8-b308-e649454b5c6c-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:30:50.605586 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.605556    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"]
Aug 04 18:30:50.608805 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:50.608781    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-765vv"]
Aug 04 18:30:51.137659 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:51.137628    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"
Aug 04 18:30:52.587367 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:52.587332    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d4e3e36-387e-40d8-b308-e649454b5c6c" path="/var/lib/kubelet/pods/1d4e3e36-387e-40d8-b308-e649454b5c6c/volumes"
Aug 04 18:30:56.114142 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:56.114088    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:56.117086 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:56.117057    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"]
Aug 04 18:30:56.169996 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:56.169961    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:56.719811 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:56.719769    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:56.821160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:56.821121    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:56.923122 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:30:56.923074    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:01.138197 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:01.138156    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5" podUID="1cdc0836-a77f-4eaf-a5d1-146870a1d356" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:31:01.751908 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:01.751870    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:05.566566 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:05.566527    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:06.598054 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:06.598014    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:11.137413 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:11.137355    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5" podUID="1cdc0836-a77f-4eaf-a5d1-146870a1d356" containerName="main" containerID="cri-o://79e640eaff021e6e57f3f66afcb23db71fa0d0a482967f080a4038a52817994e" gracePeriod=45
Aug 04 18:31:11.138402 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:11.138369    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5" podUID="1cdc0836-a77f-4eaf-a5d1-146870a1d356" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:31:11.387986 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:11.387922    2576 generic.go:356] "Generic (PLEG): container finished" podID="1cdc0836-a77f-4eaf-a5d1-146870a1d356" containerID="79e640eaff021e6e57f3f66afcb23db71fa0d0a482967f080a4038a52817994e" exitCode=0
Aug 04 18:31:11.388090 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:11.387993    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5" event={"ID":"1cdc0836-a77f-4eaf-a5d1-146870a1d356","Type":"ContainerDied","Data":"79e640eaff021e6e57f3f66afcb23db71fa0d0a482967f080a4038a52817994e"}
Aug 04 18:31:11.388090 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:11.388046    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5" event={"ID":"1cdc0836-a77f-4eaf-a5d1-146870a1d356","Type":"ContainerDied","Data":"59aeeb509d303a257ca31fd262884e27f6f5b75eca63afddd909dfd3650ff879"}
Aug 04 18:31:11.388090 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:11.388065    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59aeeb509d303a257ca31fd262884e27f6f5b75eca63afddd909dfd3650ff879"
Aug 04 18:31:11.392392 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:11.392372    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"
Aug 04 18:31:11.436663 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:11.436629    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdc0836-a77f-4eaf-a5d1-146870a1d356-tls-certs\") pod \"1cdc0836-a77f-4eaf-a5d1-146870a1d356\" (UID: \"1cdc0836-a77f-4eaf-a5d1-146870a1d356\") "
Aug 04 18:31:11.436874 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:11.436753    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8b4w\" (UniqueName: \"kubernetes.io/projected/1cdc0836-a77f-4eaf-a5d1-146870a1d356-kube-api-access-t8b4w\") pod \"1cdc0836-a77f-4eaf-a5d1-146870a1d356\" (UID: \"1cdc0836-a77f-4eaf-a5d1-146870a1d356\") "
Aug 04 18:31:11.438946 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:11.438921    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cdc0836-a77f-4eaf-a5d1-146870a1d356-kube-api-access-t8b4w" (OuterVolumeSpecName: "kube-api-access-t8b4w") pod "1cdc0836-a77f-4eaf-a5d1-146870a1d356" (UID: "1cdc0836-a77f-4eaf-a5d1-146870a1d356"). InnerVolumeSpecName "kube-api-access-t8b4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:31:11.439075 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:11.439011    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cdc0836-a77f-4eaf-a5d1-146870a1d356-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cdc0836-a77f-4eaf-a5d1-146870a1d356" (UID: "1cdc0836-a77f-4eaf-a5d1-146870a1d356"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:31:11.538121 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:11.538084    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdc0836-a77f-4eaf-a5d1-146870a1d356-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:31:11.538281 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:11.538146    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8b4w\" (UniqueName: \"kubernetes.io/projected/1cdc0836-a77f-4eaf-a5d1-146870a1d356-kube-api-access-t8b4w\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:31:12.392628 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:12.392598    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"
Aug 04 18:31:12.414336 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:12.414310    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"]
Aug 04 18:31:12.420126 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:12.420090    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scher54k5"]
Aug 04 18:31:12.587607 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:12.587572    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cdc0836-a77f-4eaf-a5d1-146870a1d356" path="/var/lib/kubelet/pods/1cdc0836-a77f-4eaf-a5d1-146870a1d356/volumes"
Aug 04 18:31:16.136697 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:16.136659    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:16.237292 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:16.237259    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:19.598143 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.598097    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"]
Aug 04 18:31:19.598513 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.598457    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4e3e36-387e-40d8-b308-e649454b5c6c" containerName="main"
Aug 04 18:31:19.598513 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.598469    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4e3e36-387e-40d8-b308-e649454b5c6c" containerName="main"
Aug 04 18:31:19.598513 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.598478    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cdc0836-a77f-4eaf-a5d1-146870a1d356" containerName="main"
Aug 04 18:31:19.598513 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.598484    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cdc0836-a77f-4eaf-a5d1-146870a1d356" containerName="main"
Aug 04 18:31:19.598647 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.598566    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cdc0836-a77f-4eaf-a5d1-146870a1d356" containerName="main"
Aug 04 18:31:19.598647 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.598575    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d4e3e36-387e-40d8-b308-e649454b5c6c" containerName="main"
Aug 04 18:31:19.601688 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.601655    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.605288 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.605263    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-sq7bp\""
Aug 04 18:31:19.605432 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.605272    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Aug 04 18:31:19.605432 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.605328    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Aug 04 18:31:19.605432 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.605318    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Aug 04 18:31:19.612283 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.612261    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"]
Aug 04 18:31:19.630459 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.630430    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:19.702830 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.702795    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-model-cache\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.703015 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.702843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnd8s\" (UniqueName: \"kubernetes.io/projected/ee1274c1-2b36-4010-95cf-23fba1f8b278-kube-api-access-vnd8s\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.703015 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.702877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-tmp-dir\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.703015 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.703000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1274c1-2b36-4010-95cf-23fba1f8b278-tls-certs\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.703158 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.703028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-home\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.703158 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.703043    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-dshm\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.803996 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.803957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-model-cache\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.804195 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.804016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnd8s\" (UniqueName: \"kubernetes.io/projected/ee1274c1-2b36-4010-95cf-23fba1f8b278-kube-api-access-vnd8s\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.804195 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.804040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-tmp-dir\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.804195 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.804146    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1274c1-2b36-4010-95cf-23fba1f8b278-tls-certs\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.804195 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.804172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-home\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.804195 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.804189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-dshm\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.804481 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.804376    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-model-cache\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.804481 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.804446    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-tmp-dir\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.804609 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.804586    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-home\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.806333 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.806316    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-dshm\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.806579 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.806562    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1274c1-2b36-4010-95cf-23fba1f8b278-tls-certs\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.812659 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.812637    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnd8s\" (UniqueName: \"kubernetes.io/projected/ee1274c1-2b36-4010-95cf-23fba1f8b278-kube-api-access-vnd8s\") pod \"decom-v2-kserve-77f867d787-v6zrz\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:19.914537 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:19.914441    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:20.068949 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:20.067416    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"]
Aug 04 18:31:20.074680 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:31:20.074655    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1274c1_2b36_4010_95cf_23fba1f8b278.slice/crio-963fd2f11550400a1878bc92fdcdf4ee71052b291893c30443fbd3272a101e1d WatchSource:0}: Error finding container 963fd2f11550400a1878bc92fdcdf4ee71052b291893c30443fbd3272a101e1d: Status 404 returned error can't find the container with id 963fd2f11550400a1878bc92fdcdf4ee71052b291893c30443fbd3272a101e1d
Aug 04 18:31:20.432265 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:20.432222    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz" event={"ID":"ee1274c1-2b36-4010-95cf-23fba1f8b278","Type":"ContainerStarted","Data":"2b57529eb3d98a97c8add49c134a43d934acc91a63ab03d1a662666ca1bece9d"}
Aug 04 18:31:20.432469 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:20.432321    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz" event={"ID":"ee1274c1-2b36-4010-95cf-23fba1f8b278","Type":"ContainerStarted","Data":"963fd2f11550400a1878bc92fdcdf4ee71052b291893c30443fbd3272a101e1d"}
Aug 04 18:31:20.452788 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:20.452714    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz" podStartSLOduration=1.45269391 podStartE2EDuration="1.45269391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:31:20.449934357 +0000 UTC m=+1880.425766073" watchObservedRunningTime="2026-08-04 18:31:20.45269391 +0000 UTC m=+1880.428525609"
Aug 04 18:31:20.915149 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:20.915075    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:23.292957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:23.292878    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:29.050361 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:29.050319    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:29.106451 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:29.106395    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:29.915506 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:29.915460    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:29.927812 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:29.927789    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:30.490991 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:30.490961    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:31:35.206692 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.206640    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"]
Aug 04 18:31:35.210311 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.210283    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.213808 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.213779    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Aug 04 18:31:35.213957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.213784    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Aug 04 18:31:35.213957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.213820    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-g5ksp\""
Aug 04 18:31:35.213957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.213838    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Aug 04 18:31:35.221454 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.221429    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"]
Aug 04 18:31:35.245157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.245100    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:35.351297 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.351249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.351484 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.351331    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.351484 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.351350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a088a8b-4342-44e7-9385-7e2389f4eec0-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.351484 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.351382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.351484 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.351452    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.351484 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.351472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z972q\" (UniqueName: \"kubernetes.io/projected/5a088a8b-4342-44e7-9385-7e2389f4eec0-kube-api-access-z972q\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.422646 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.422608    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"]
Aug 04 18:31:35.426041 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.426021    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"
Aug 04 18:31:35.428586 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.428564    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-lx45r\""
Aug 04 18:31:35.446171 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.446141    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"]
Aug 04 18:31:35.452613 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.452584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.452751 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.452631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z972q\" (UniqueName: \"kubernetes.io/projected/5a088a8b-4342-44e7-9385-7e2389f4eec0-kube-api-access-z972q\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.452751 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.452680    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.452864 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.452763    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.452864 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.452788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a088a8b-4342-44e7-9385-7e2389f4eec0-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.452864 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.452838    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.453178 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.453139    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.453178 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.453157    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.453336 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.453207    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.455042 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.455023    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.455478 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.455459    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a088a8b-4342-44e7-9385-7e2389f4eec0-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.465905 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.465849    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z972q\" (UniqueName: \"kubernetes.io/projected/5a088a8b-4342-44e7-9385-7e2389f4eec0-kube-api-access-z972q\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.522151 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.522094    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:35.554306 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.554261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d15ab2-320d-4b9f-b0ac-a95ea2499c60-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm\" (UID: \"e8d15ab2-320d-4b9f-b0ac-a95ea2499c60\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"
Aug 04 18:31:35.554470 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.554351    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn5tb\" (UniqueName: \"kubernetes.io/projected/e8d15ab2-320d-4b9f-b0ac-a95ea2499c60-kube-api-access-jn5tb\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm\" (UID: \"e8d15ab2-320d-4b9f-b0ac-a95ea2499c60\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"
Aug 04 18:31:35.655685 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.655649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d15ab2-320d-4b9f-b0ac-a95ea2499c60-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm\" (UID: \"e8d15ab2-320d-4b9f-b0ac-a95ea2499c60\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"
Aug 04 18:31:35.655851 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.655695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn5tb\" (UniqueName: \"kubernetes.io/projected/e8d15ab2-320d-4b9f-b0ac-a95ea2499c60-kube-api-access-jn5tb\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm\" (UID: \"e8d15ab2-320d-4b9f-b0ac-a95ea2499c60\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"
Aug 04 18:31:35.658021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.658000    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d15ab2-320d-4b9f-b0ac-a95ea2499c60-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm\" (UID: \"e8d15ab2-320d-4b9f-b0ac-a95ea2499c60\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"
Aug 04 18:31:35.663257 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.663193    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"]
Aug 04 18:31:35.665726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.665698    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn5tb\" (UniqueName: \"kubernetes.io/projected/e8d15ab2-320d-4b9f-b0ac-a95ea2499c60-kube-api-access-jn5tb\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm\" (UID: \"e8d15ab2-320d-4b9f-b0ac-a95ea2499c60\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"
Aug 04 18:31:35.736827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.736694    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"
Aug 04 18:31:35.877208 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:35.877183    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"]
Aug 04 18:31:35.879735 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:31:35.879705    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d15ab2_320d_4b9f_b0ac_a95ea2499c60.slice/crio-5ff1b146cbfc88d4ed9af9b61621800d2f997a19d0d3b5082a5c0244038a6e0b WatchSource:0}: Error finding container 5ff1b146cbfc88d4ed9af9b61621800d2f997a19d0d3b5082a5c0244038a6e0b: Status 404 returned error can't find the container with id 5ff1b146cbfc88d4ed9af9b61621800d2f997a19d0d3b5082a5c0244038a6e0b
Aug 04 18:31:36.509651 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:36.509611    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x" event={"ID":"5a088a8b-4342-44e7-9385-7e2389f4eec0","Type":"ContainerStarted","Data":"15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24"}
Aug 04 18:31:36.510142 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:36.509681    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x" event={"ID":"5a088a8b-4342-44e7-9385-7e2389f4eec0","Type":"ContainerStarted","Data":"bc440b24a7fc6557af76116c632c82c6627ae721f353d0775b169c9021b44551"}
Aug 04 18:31:36.511071 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:36.511047    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm" event={"ID":"e8d15ab2-320d-4b9f-b0ac-a95ea2499c60","Type":"ContainerStarted","Data":"01f4cf3d8ed5154caf6ffdf625bd3d7f899f704a29e92ddcb66526dd84eeb789"}
Aug 04 18:31:36.511211 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:36.511095    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm" event={"ID":"e8d15ab2-320d-4b9f-b0ac-a95ea2499c60","Type":"ContainerStarted","Data":"5ff1b146cbfc88d4ed9af9b61621800d2f997a19d0d3b5082a5c0244038a6e0b"}
Aug 04 18:31:36.511211 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:36.511148    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"
Aug 04 18:31:36.523127 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:36.523088    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:36.528604 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:36.528567    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x" podStartSLOduration=1.528553402 podStartE2EDuration="1.528553402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:31:36.527081516 +0000 UTC m=+1896.502913215" watchObservedRunningTime="2026-08-04 18:31:36.528553402 +0000 UTC m=+1896.504385099"
Aug 04 18:31:36.547989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:36.547945    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm" podStartSLOduration=1.547930364 podStartE2EDuration="1.547930364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:31:36.545888729 +0000 UTC m=+1896.521720430" watchObservedRunningTime="2026-08-04 18:31:36.547930364 +0000 UTC m=+1896.523762051"
Aug 04 18:31:37.632683 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:37.632650    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:39.396196 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:39.396162    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:45.355647 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:45.355606    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:45.442655 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:45.442610    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:45.522615 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:45.522564    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:45.535064 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:45.535038    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:45.567392 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:45.567364    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:31:49.548741 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:49.548679    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:49.648938 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:31:49.648901    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:00.752962 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:00.752916    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:00.854226 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:00.854188    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:00.953885 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:00.953850    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:07.518313 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:07.518284    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"
Aug 04 18:32:13.128982 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:13.128945    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:21.439008 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:21.438968    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:21.719263 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:21.719169    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:30.065411 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:30.065374    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:33.872765 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:33.872730    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"]
Aug 04 18:32:33.893363 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:33.893331    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:48.892833 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:48.892764    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz" podUID="ee1274c1-2b36-4010-95cf-23fba1f8b278" containerName="main" containerID="cri-o://2b57529eb3d98a97c8add49c134a43d934acc91a63ab03d1a662666ca1bece9d" gracePeriod=45
Aug 04 18:32:49.149025 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.148959    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:32:49.168268 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.168234    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:49.257970 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.257941    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-dshm\") pod \"ee1274c1-2b36-4010-95cf-23fba1f8b278\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") "
Aug 04 18:32:49.258169 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.258015    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-tmp-dir\") pod \"ee1274c1-2b36-4010-95cf-23fba1f8b278\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") "
Aug 04 18:32:49.258169 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.258047    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1274c1-2b36-4010-95cf-23fba1f8b278-tls-certs\") pod \"ee1274c1-2b36-4010-95cf-23fba1f8b278\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") "
Aug 04 18:32:49.258169 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.258069    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnd8s\" (UniqueName: \"kubernetes.io/projected/ee1274c1-2b36-4010-95cf-23fba1f8b278-kube-api-access-vnd8s\") pod \"ee1274c1-2b36-4010-95cf-23fba1f8b278\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") "
Aug 04 18:32:49.258169 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.258093    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-home\") pod \"ee1274c1-2b36-4010-95cf-23fba1f8b278\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") "
Aug 04 18:32:49.258169 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.258147    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-model-cache\") pod \"ee1274c1-2b36-4010-95cf-23fba1f8b278\" (UID: \"ee1274c1-2b36-4010-95cf-23fba1f8b278\") "
Aug 04 18:32:49.258459 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.258295    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee1274c1-2b36-4010-95cf-23fba1f8b278" (UID: "ee1274c1-2b36-4010-95cf-23fba1f8b278"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:32:49.258459 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.258411    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-home" (OuterVolumeSpecName: "home") pod "ee1274c1-2b36-4010-95cf-23fba1f8b278" (UID: "ee1274c1-2b36-4010-95cf-23fba1f8b278"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:32:49.258459 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.258435    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee1274c1-2b36-4010-95cf-23fba1f8b278" (UID: "ee1274c1-2b36-4010-95cf-23fba1f8b278"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:32:49.258634 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.258531    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:32:49.258634 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.258545    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:32:49.258634 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.258554    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:32:49.260544 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.260520    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-dshm" (OuterVolumeSpecName: "dshm") pod "ee1274c1-2b36-4010-95cf-23fba1f8b278" (UID: "ee1274c1-2b36-4010-95cf-23fba1f8b278"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:32:49.260673 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.260523    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1274c1-2b36-4010-95cf-23fba1f8b278-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee1274c1-2b36-4010-95cf-23fba1f8b278" (UID: "ee1274c1-2b36-4010-95cf-23fba1f8b278"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:32:49.260673 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.260642    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1274c1-2b36-4010-95cf-23fba1f8b278-kube-api-access-vnd8s" (OuterVolumeSpecName: "kube-api-access-vnd8s") pod "ee1274c1-2b36-4010-95cf-23fba1f8b278" (UID: "ee1274c1-2b36-4010-95cf-23fba1f8b278"). InnerVolumeSpecName "kube-api-access-vnd8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:32:49.278796 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.278760    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:49.359201 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.359166    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1274c1-2b36-4010-95cf-23fba1f8b278-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:32:49.359201 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.359195    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnd8s\" (UniqueName: \"kubernetes.io/projected/ee1274c1-2b36-4010-95cf-23fba1f8b278-kube-api-access-vnd8s\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:32:49.359201 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.359207    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1274c1-2b36-4010-95cf-23fba1f8b278-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:32:49.402745 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.402662    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:49.865763 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.865731    2576 generic.go:356] "Generic (PLEG): container finished" podID="ee1274c1-2b36-4010-95cf-23fba1f8b278" containerID="2b57529eb3d98a97c8add49c134a43d934acc91a63ab03d1a662666ca1bece9d" exitCode=0
Aug 04 18:32:49.865941 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.865784    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"
Aug 04 18:32:49.865941 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.865787    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz" event={"ID":"ee1274c1-2b36-4010-95cf-23fba1f8b278","Type":"ContainerDied","Data":"2b57529eb3d98a97c8add49c134a43d934acc91a63ab03d1a662666ca1bece9d"}
Aug 04 18:32:49.865941 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.865896    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz" event={"ID":"ee1274c1-2b36-4010-95cf-23fba1f8b278","Type":"ContainerDied","Data":"963fd2f11550400a1878bc92fdcdf4ee71052b291893c30443fbd3272a101e1d"}
Aug 04 18:32:49.865941 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.865921    2576 scope.go:117] "RemoveContainer" containerID="2b57529eb3d98a97c8add49c134a43d934acc91a63ab03d1a662666ca1bece9d"
Aug 04 18:32:49.875030 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.875009    2576 scope.go:117] "RemoveContainer" containerID="2b57529eb3d98a97c8add49c134a43d934acc91a63ab03d1a662666ca1bece9d"
Aug 04 18:32:49.875305 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:32:49.875285    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b57529eb3d98a97c8add49c134a43d934acc91a63ab03d1a662666ca1bece9d\": container with ID starting with 2b57529eb3d98a97c8add49c134a43d934acc91a63ab03d1a662666ca1bece9d not found: ID does not exist" containerID="2b57529eb3d98a97c8add49c134a43d934acc91a63ab03d1a662666ca1bece9d"
Aug 04 18:32:49.875368 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.875315    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b57529eb3d98a97c8add49c134a43d934acc91a63ab03d1a662666ca1bece9d"} err="failed to get container status \"2b57529eb3d98a97c8add49c134a43d934acc91a63ab03d1a662666ca1bece9d\": rpc error: code = NotFound desc = could not find container \"2b57529eb3d98a97c8add49c134a43d934acc91a63ab03d1a662666ca1bece9d\": container with ID starting with 2b57529eb3d98a97c8add49c134a43d934acc91a63ab03d1a662666ca1bece9d not found: ID does not exist"
Aug 04 18:32:49.888858 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.888832    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"]
Aug 04 18:32:49.893847 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:49.893824    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-v6zrz"]
Aug 04 18:32:50.582764 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:50.582734    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee1274c1-2b36-4010-95cf-23fba1f8b278" path="/var/lib/kubelet/pods/ee1274c1-2b36-4010-95cf-23fba1f8b278/volumes"
Aug 04 18:32:54.239589 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:32:54.239502    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:08.301043 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:33:08.301004    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:15.631474 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:33:15.631417    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:31.225884 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:33:31.225844    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:33.539175 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:33:33.539130    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:36.872335 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:33:36.872299    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:36.948214 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:33:36.948174    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:53.248013 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:33:53.247945    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:53.380185 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:33:53.380152    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:00.362881 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:00.362843    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:00.459512 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:00.459480    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:00.561607 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:00.561573    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:01.388336 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:01.388297    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:05.429393 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:05.429354    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:26.353789 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:26.353709    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:33.344008 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:33.343970    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:42.233932 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:42.233894    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:43.005199 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:43.005158    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:49.127435 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:49.127390    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:49.237285 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:49.237249    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:58.224051 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:58.224014    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:59.472851 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.472813    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:59.498782 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.498743    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"]
Aug 04 18:34:59.533293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.533258    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"]
Aug 04 18:34:59.549149 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:34:59.549081    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:34:59.549149 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:34:59.549149 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:34:59.549149 ip-10-0-133-187 kubenswrapper[2576]:  > containerID="15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24" cmd=["/bin/sleep","15"]
Aug 04 18:34:59.549568 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.549547    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x" message="" err=<
Aug 04 18:34:59.549568 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:34:59.549568 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:34:59.549568 ip-10-0-133-187 kubenswrapper[2576]:  >
Aug 04 18:34:59.549810 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:34:59.549792    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:34:59.549810 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:34:59.549810 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:34:59.549810 ip-10-0-133-187 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x" podUID="5a088a8b-4342-44e7-9385-7e2389f4eec0" containerName="main" containerID="cri-o://15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24"
Aug 04 18:34:59.550120 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.550077    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x" podUID="5a088a8b-4342-44e7-9385-7e2389f4eec0" containerName="main" containerID="cri-o://15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24" gracePeriod=60
Aug 04 18:34:59.557374 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.557342    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x" podUID="5a088a8b-4342-44e7-9385-7e2389f4eec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Aug 04 18:34:59.610281 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.610250    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:59.907798 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.907771    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:34:59.950902 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.950871    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a088a8b-4342-44e7-9385-7e2389f4eec0-tls-certs\") pod \"5a088a8b-4342-44e7-9385-7e2389f4eec0\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") "
Aug 04 18:34:59.950902 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.950905    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z972q\" (UniqueName: \"kubernetes.io/projected/5a088a8b-4342-44e7-9385-7e2389f4eec0-kube-api-access-z972q\") pod \"5a088a8b-4342-44e7-9385-7e2389f4eec0\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") "
Aug 04 18:34:59.951208 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.950936    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-home\") pod \"5a088a8b-4342-44e7-9385-7e2389f4eec0\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") "
Aug 04 18:34:59.951208 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.950950    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-dshm\") pod \"5a088a8b-4342-44e7-9385-7e2389f4eec0\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") "
Aug 04 18:34:59.951208 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.951077    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-model-cache\") pod \"5a088a8b-4342-44e7-9385-7e2389f4eec0\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") "
Aug 04 18:34:59.951208 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.951102    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-tmp-dir\") pod \"5a088a8b-4342-44e7-9385-7e2389f4eec0\" (UID: \"5a088a8b-4342-44e7-9385-7e2389f4eec0\") "
Aug 04 18:34:59.951437 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.951232    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-home" (OuterVolumeSpecName: "home") pod "5a088a8b-4342-44e7-9385-7e2389f4eec0" (UID: "5a088a8b-4342-44e7-9385-7e2389f4eec0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:34:59.951437 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.951347    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a088a8b-4342-44e7-9385-7e2389f4eec0" (UID: "5a088a8b-4342-44e7-9385-7e2389f4eec0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:34:59.951437 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.951407    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a088a8b-4342-44e7-9385-7e2389f4eec0" (UID: "5a088a8b-4342-44e7-9385-7e2389f4eec0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:34:59.951602 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.951491    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:34:59.951602 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.951510    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:34:59.951602 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.951523    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:34:59.953560 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.953527    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-dshm" (OuterVolumeSpecName: "dshm") pod "5a088a8b-4342-44e7-9385-7e2389f4eec0" (UID: "5a088a8b-4342-44e7-9385-7e2389f4eec0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:34:59.953842 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.953824    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a088a8b-4342-44e7-9385-7e2389f4eec0-kube-api-access-z972q" (OuterVolumeSpecName: "kube-api-access-z972q") pod "5a088a8b-4342-44e7-9385-7e2389f4eec0" (UID: "5a088a8b-4342-44e7-9385-7e2389f4eec0"). InnerVolumeSpecName "kube-api-access-z972q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:34:59.953930 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:34:59.953915    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a088a8b-4342-44e7-9385-7e2389f4eec0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a088a8b-4342-44e7-9385-7e2389f4eec0" (UID: "5a088a8b-4342-44e7-9385-7e2389f4eec0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:35:00.052830 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:00.052788    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a088a8b-4342-44e7-9385-7e2389f4eec0-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:35:00.052830 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:00.052823    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z972q\" (UniqueName: \"kubernetes.io/projected/5a088a8b-4342-44e7-9385-7e2389f4eec0-kube-api-access-z972q\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:35:00.052830 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:00.052833    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a088a8b-4342-44e7-9385-7e2389f4eec0-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:35:00.502323 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:00.502235    2576 generic.go:356] "Generic (PLEG): container finished" podID="5a088a8b-4342-44e7-9385-7e2389f4eec0" containerID="15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24" exitCode=0
Aug 04 18:35:00.502323 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:00.502281    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"
Aug 04 18:35:00.502776 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:00.502315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x" event={"ID":"5a088a8b-4342-44e7-9385-7e2389f4eec0","Type":"ContainerDied","Data":"15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24"}
Aug 04 18:35:00.502776 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:00.502376    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x" event={"ID":"5a088a8b-4342-44e7-9385-7e2389f4eec0","Type":"ContainerDied","Data":"bc440b24a7fc6557af76116c632c82c6627ae721f353d0775b169c9021b44551"}
Aug 04 18:35:00.502776 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:00.502399    2576 scope.go:117] "RemoveContainer" containerID="15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24"
Aug 04 18:35:00.511855 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:00.511834    2576 scope.go:117] "RemoveContainer" containerID="15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24"
Aug 04 18:35:00.512096 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:35:00.512077    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24\": container with ID starting with 15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24 not found: ID does not exist" containerID="15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24"
Aug 04 18:35:00.512156 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:00.512105    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24"} err="failed to get container status \"15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24\": rpc error: code = NotFound desc = could not find container \"15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24\": container with ID starting with 15ad3aad0afa2c50a1085139aebc530a98b30debff06b4b36530e9d0dd679f24 not found: ID does not exist"
Aug 04 18:35:00.530322 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:00.530294    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"]
Aug 04 18:35:00.536752 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:00.536724    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xx78x"]
Aug 04 18:35:00.583140 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:00.583089    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a088a8b-4342-44e7-9385-7e2389f4eec0" path="/var/lib/kubelet/pods/5a088a8b-4342-44e7-9385-7e2389f4eec0/volumes"
Aug 04 18:35:07.518054 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:07.518014    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm" podUID="e8d15ab2-320d-4b9f-b0ac-a95ea2499c60" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:35:14.525878 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:14.525832    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm" podUID="e8d15ab2-320d-4b9f-b0ac-a95ea2499c60" containerName="main" containerID="cri-o://01f4cf3d8ed5154caf6ffdf625bd3d7f899f704a29e92ddcb66526dd84eeb789" gracePeriod=45
Aug 04 18:35:14.779468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:14.779407    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"
Aug 04 18:35:14.895830 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:14.895795    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn5tb\" (UniqueName: \"kubernetes.io/projected/e8d15ab2-320d-4b9f-b0ac-a95ea2499c60-kube-api-access-jn5tb\") pod \"e8d15ab2-320d-4b9f-b0ac-a95ea2499c60\" (UID: \"e8d15ab2-320d-4b9f-b0ac-a95ea2499c60\") "
Aug 04 18:35:14.896036 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:14.895916    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d15ab2-320d-4b9f-b0ac-a95ea2499c60-tls-certs\") pod \"e8d15ab2-320d-4b9f-b0ac-a95ea2499c60\" (UID: \"e8d15ab2-320d-4b9f-b0ac-a95ea2499c60\") "
Aug 04 18:35:14.897940 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:14.897910    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d15ab2-320d-4b9f-b0ac-a95ea2499c60-kube-api-access-jn5tb" (OuterVolumeSpecName: "kube-api-access-jn5tb") pod "e8d15ab2-320d-4b9f-b0ac-a95ea2499c60" (UID: "e8d15ab2-320d-4b9f-b0ac-a95ea2499c60"). InnerVolumeSpecName "kube-api-access-jn5tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:35:14.898055 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:14.897947    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d15ab2-320d-4b9f-b0ac-a95ea2499c60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8d15ab2-320d-4b9f-b0ac-a95ea2499c60" (UID: "e8d15ab2-320d-4b9f-b0ac-a95ea2499c60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:35:14.997206 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:14.997158    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d15ab2-320d-4b9f-b0ac-a95ea2499c60-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:35:14.997206 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:14.997202    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn5tb\" (UniqueName: \"kubernetes.io/projected/e8d15ab2-320d-4b9f-b0ac-a95ea2499c60-kube-api-access-jn5tb\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:35:15.573784 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:15.573746    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8d15ab2-320d-4b9f-b0ac-a95ea2499c60" containerID="01f4cf3d8ed5154caf6ffdf625bd3d7f899f704a29e92ddcb66526dd84eeb789" exitCode=0
Aug 04 18:35:15.574302 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:15.573788    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"
Aug 04 18:35:15.574302 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:15.573828    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm" event={"ID":"e8d15ab2-320d-4b9f-b0ac-a95ea2499c60","Type":"ContainerDied","Data":"01f4cf3d8ed5154caf6ffdf625bd3d7f899f704a29e92ddcb66526dd84eeb789"}
Aug 04 18:35:15.574302 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:15.573880    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm" event={"ID":"e8d15ab2-320d-4b9f-b0ac-a95ea2499c60","Type":"ContainerDied","Data":"5ff1b146cbfc88d4ed9af9b61621800d2f997a19d0d3b5082a5c0244038a6e0b"}
Aug 04 18:35:15.574302 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:15.573908    2576 scope.go:117] "RemoveContainer" containerID="01f4cf3d8ed5154caf6ffdf625bd3d7f899f704a29e92ddcb66526dd84eeb789"
Aug 04 18:35:15.583222 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:15.583205    2576 scope.go:117] "RemoveContainer" containerID="01f4cf3d8ed5154caf6ffdf625bd3d7f899f704a29e92ddcb66526dd84eeb789"
Aug 04 18:35:15.583475 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:35:15.583457    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f4cf3d8ed5154caf6ffdf625bd3d7f899f704a29e92ddcb66526dd84eeb789\": container with ID starting with 01f4cf3d8ed5154caf6ffdf625bd3d7f899f704a29e92ddcb66526dd84eeb789 not found: ID does not exist" containerID="01f4cf3d8ed5154caf6ffdf625bd3d7f899f704a29e92ddcb66526dd84eeb789"
Aug 04 18:35:15.583529 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:15.583485    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f4cf3d8ed5154caf6ffdf625bd3d7f899f704a29e92ddcb66526dd84eeb789"} err="failed to get container status \"01f4cf3d8ed5154caf6ffdf625bd3d7f899f704a29e92ddcb66526dd84eeb789\": rpc error: code = NotFound desc = could not find container \"01f4cf3d8ed5154caf6ffdf625bd3d7f899f704a29e92ddcb66526dd84eeb789\": container with ID starting with 01f4cf3d8ed5154caf6ffdf625bd3d7f899f704a29e92ddcb66526dd84eeb789 not found: ID does not exist"
Aug 04 18:35:15.605372 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:15.605342    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"]
Aug 04 18:35:15.610885 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:15.610858    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sches5fnm"]
Aug 04 18:35:16.582436 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:16.582402    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8d15ab2-320d-4b9f-b0ac-a95ea2499c60" path="/var/lib/kubelet/pods/e8d15ab2-320d-4b9f-b0ac-a95ea2499c60/volumes"
Aug 04 18:35:20.424834 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:20.424797    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:20.513791 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:20.513752    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:27.403532 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:27.403497    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:33.314997 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:33.314958    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:33.416204 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:33.416163    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:42.204819 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.204777    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"]
Aug 04 18:35:42.205365 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.205344    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a088a8b-4342-44e7-9385-7e2389f4eec0" containerName="main"
Aug 04 18:35:42.205448 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.205370    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a088a8b-4342-44e7-9385-7e2389f4eec0" containerName="main"
Aug 04 18:35:42.205448 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.205398    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1274c1-2b36-4010-95cf-23fba1f8b278" containerName="main"
Aug 04 18:35:42.205448 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.205407    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1274c1-2b36-4010-95cf-23fba1f8b278" containerName="main"
Aug 04 18:35:42.205448 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.205424    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d15ab2-320d-4b9f-b0ac-a95ea2499c60" containerName="main"
Aug 04 18:35:42.205448 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.205434    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d15ab2-320d-4b9f-b0ac-a95ea2499c60" containerName="main"
Aug 04 18:35:42.205684 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.205528    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8d15ab2-320d-4b9f-b0ac-a95ea2499c60" containerName="main"
Aug 04 18:35:42.205684 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.205542    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1274c1-2b36-4010-95cf-23fba1f8b278" containerName="main"
Aug 04 18:35:42.205684 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.205554    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a088a8b-4342-44e7-9385-7e2389f4eec0" containerName="main"
Aug 04 18:35:42.208507 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.208485    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.211049 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.211018    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Aug 04 18:35:42.211176 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.211148    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Aug 04 18:35:42.212022 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.212008    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-7zpwf\""
Aug 04 18:35:42.212241 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.212226    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Aug 04 18:35:42.217735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.217712    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"]
Aug 04 18:35:42.301414 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.301382    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:42.346734 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.346698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.346734 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.346737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.346967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.346776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.346967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.346810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.346967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.346837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.346967 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.346871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtgqg\" (UniqueName: \"kubernetes.io/projected/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-kube-api-access-gtgqg\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.447536 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.447503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.447706 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.447543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.447706 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.447584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.447884 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.447863    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgqg\" (UniqueName: \"kubernetes.io/projected/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-kube-api-access-gtgqg\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.447948 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.447924    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.447992 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.447938    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.448043 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.448000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.448091 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.448039    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.448304 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.448278    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.449993 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.449966    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.450282 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.450261    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.465020 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.464953    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtgqg\" (UniqueName: \"kubernetes.io/projected/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-kube-api-access-gtgqg\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.519456 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.519418    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:42.597030 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.596995    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"]
Aug 04 18:35:42.601369 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.601328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"
Aug 04 18:35:42.604105 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.604079    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-lqx4j\""
Aug 04 18:35:42.608273 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.608239    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"]
Aug 04 18:35:42.624535 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.624450    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:42.650173 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.650008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb70ec4d-80d5-4856-88f3-ffc8d420b628-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d\" (UID: \"cb70ec4d-80d5-4856-88f3-ffc8d420b628\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"
Aug 04 18:35:42.650173 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.650059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f96bz\" (UniqueName: \"kubernetes.io/projected/cb70ec4d-80d5-4856-88f3-ffc8d420b628-kube-api-access-f96bz\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d\" (UID: \"cb70ec4d-80d5-4856-88f3-ffc8d420b628\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"
Aug 04 18:35:42.676273 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.676247    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"]
Aug 04 18:35:42.679042 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:35:42.679003    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bb2f25_9952_4bef_8c2d_5bf09f0061b2.slice/crio-2cac863c6b551899253c58ffa331c34b3f30e6e2f38f9f113b2730416a5a7994 WatchSource:0}: Error finding container 2cac863c6b551899253c58ffa331c34b3f30e6e2f38f9f113b2730416a5a7994: Status 404 returned error can't find the container with id 2cac863c6b551899253c58ffa331c34b3f30e6e2f38f9f113b2730416a5a7994
Aug 04 18:35:42.701894 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.701860    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd" event={"ID":"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2","Type":"ContainerStarted","Data":"2cac863c6b551899253c58ffa331c34b3f30e6e2f38f9f113b2730416a5a7994"}
Aug 04 18:35:42.751206 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.751181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb70ec4d-80d5-4856-88f3-ffc8d420b628-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d\" (UID: \"cb70ec4d-80d5-4856-88f3-ffc8d420b628\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"
Aug 04 18:35:42.751313 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.751213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f96bz\" (UniqueName: \"kubernetes.io/projected/cb70ec4d-80d5-4856-88f3-ffc8d420b628-kube-api-access-f96bz\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d\" (UID: \"cb70ec4d-80d5-4856-88f3-ffc8d420b628\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"
Aug 04 18:35:42.753468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.753447    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb70ec4d-80d5-4856-88f3-ffc8d420b628-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d\" (UID: \"cb70ec4d-80d5-4856-88f3-ffc8d420b628\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"
Aug 04 18:35:42.763591 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.763569    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f96bz\" (UniqueName: \"kubernetes.io/projected/cb70ec4d-80d5-4856-88f3-ffc8d420b628-kube-api-access-f96bz\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d\" (UID: \"cb70ec4d-80d5-4856-88f3-ffc8d420b628\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"
Aug 04 18:35:42.916291 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:42.916248    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"
Aug 04 18:35:43.051317 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:43.051291    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"]
Aug 04 18:35:43.053439 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:35:43.053412    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb70ec4d_80d5_4856_88f3_ffc8d420b628.slice/crio-62e7a167e086d9822093ea85e1d0809b88705df9621dfdb7964187b7eac5cccc WatchSource:0}: Error finding container 62e7a167e086d9822093ea85e1d0809b88705df9621dfdb7964187b7eac5cccc: Status 404 returned error can't find the container with id 62e7a167e086d9822093ea85e1d0809b88705df9621dfdb7964187b7eac5cccc
Aug 04 18:35:43.709424 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:43.709368    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd" event={"ID":"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2","Type":"ContainerStarted","Data":"73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17"}
Aug 04 18:35:43.710962 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:43.710933    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d" event={"ID":"cb70ec4d-80d5-4856-88f3-ffc8d420b628","Type":"ContainerStarted","Data":"e89dae5e74cbff87c96b2d0d10a767dab79b7cb65fe5a358ea32b6d54a69bde5"}
Aug 04 18:35:43.711144 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:43.710979    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d" event={"ID":"cb70ec4d-80d5-4856-88f3-ffc8d420b628","Type":"ContainerStarted","Data":"62e7a167e086d9822093ea85e1d0809b88705df9621dfdb7964187b7eac5cccc"}
Aug 04 18:35:43.711144 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:43.711065    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"
Aug 04 18:35:43.729158 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:43.729093    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd" podStartSLOduration=1.7290737489999999 podStartE2EDuration="1.729073749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:35:43.726437027 +0000 UTC m=+2143.702268723" watchObservedRunningTime="2026-08-04 18:35:43.729073749 +0000 UTC m=+2143.704905447"
Aug 04 18:35:43.747894 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:43.747842    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d" podStartSLOduration=1.7478273899999999 podStartE2EDuration="1.74782739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:35:43.746055545 +0000 UTC m=+2143.721887249" watchObservedRunningTime="2026-08-04 18:35:43.74782739 +0000 UTC m=+2143.723659087"
Aug 04 18:35:44.520086 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:44.520054    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:52.520440 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:52.520347    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:52.533527 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:52.533503    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:35:52.764453 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:35:52.764424    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:36:01.422507 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:01.422476    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:10.073273 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:10.073227    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:11.809913 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:11.809872    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:11.909040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:11.909005    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:12.008228 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:12.008194    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:12.635970 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:12.635933    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:13.555389 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:13.555353    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:14.717868 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:14.717836    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"
Aug 04 18:36:16.881074 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:16.881036    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:31.538926 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:31.538863    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:32.452004 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:32.451963    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:53.823249 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:53.823209    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:55.286621 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:55.286585    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:59.419828 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:59.419793    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:59.482292 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:36:59.482258    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:37:00.223536 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:00.223494    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:37:00.784491 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:00.784455    2576 scope.go:117] "RemoveContainer" containerID="79e640eaff021e6e57f3f66afcb23db71fa0d0a482967f080a4038a52817994e"
Aug 04 18:37:21.521239 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:21.521201    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:37:22.125218 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:22.125184    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:37:29.371640 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:29.371590    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:37:29.447468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:29.447439    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:37:49.316288 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.316251    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"]
Aug 04 18:37:49.318340 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.318306    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"]
Aug 04 18:37:49.319039 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.319007    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:37:49.327869 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:37:49.327826    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:37:49.327869 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:37:49.327869 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:37:49.327869 ip-10-0-133-187 kubenswrapper[2576]:  > containerID="73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17" cmd=["/bin/sleep","15"]
Aug 04 18:37:49.328177 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.327890    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd" message="" err=<
Aug 04 18:37:49.328177 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:37:49.328177 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:37:49.328177 ip-10-0-133-187 kubenswrapper[2576]:  >
Aug 04 18:37:49.328177 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:37:49.327906    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:37:49.328177 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:37:49.328177 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 18:37:49.328177 ip-10-0-133-187 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd" podUID="f4bb2f25-9952-4bef-8c2d-5bf09f0061b2" containerName="main" containerID="cri-o://73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17"
Aug 04 18:37:49.328177 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.327957    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd" podUID="f4bb2f25-9952-4bef-8c2d-5bf09f0061b2" containerName="main" containerID="cri-o://73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17" gracePeriod=60
Aug 04 18:37:49.379135 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.379078    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:37:49.685545 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.685518    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:37:49.810746 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.810713    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-home\") pod \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") "
Aug 04 18:37:49.810938 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.810753    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-dshm\") pod \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") "
Aug 04 18:37:49.810938 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.810777    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-tls-certs\") pod \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") "
Aug 04 18:37:49.810938 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.810814    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-model-cache\") pod \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") "
Aug 04 18:37:49.810938 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.810885    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtgqg\" (UniqueName: \"kubernetes.io/projected/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-kube-api-access-gtgqg\") pod \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") "
Aug 04 18:37:49.810938 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.810913    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-tmp-dir\") pod \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\" (UID: \"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2\") "
Aug 04 18:37:49.811280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.811023    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-home" (OuterVolumeSpecName: "home") pod "f4bb2f25-9952-4bef-8c2d-5bf09f0061b2" (UID: "f4bb2f25-9952-4bef-8c2d-5bf09f0061b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:37:49.811280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.811164    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4bb2f25-9952-4bef-8c2d-5bf09f0061b2" (UID: "f4bb2f25-9952-4bef-8c2d-5bf09f0061b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:37:49.811280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.811234    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:37:49.811280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.811249    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:37:49.811430 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.811306    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4bb2f25-9952-4bef-8c2d-5bf09f0061b2" (UID: "f4bb2f25-9952-4bef-8c2d-5bf09f0061b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:37:49.813055 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.813033    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-dshm" (OuterVolumeSpecName: "dshm") pod "f4bb2f25-9952-4bef-8c2d-5bf09f0061b2" (UID: "f4bb2f25-9952-4bef-8c2d-5bf09f0061b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:37:49.813175 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.813102    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4bb2f25-9952-4bef-8c2d-5bf09f0061b2" (UID: "f4bb2f25-9952-4bef-8c2d-5bf09f0061b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:37:49.813175 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.813128    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-kube-api-access-gtgqg" (OuterVolumeSpecName: "kube-api-access-gtgqg") pod "f4bb2f25-9952-4bef-8c2d-5bf09f0061b2" (UID: "f4bb2f25-9952-4bef-8c2d-5bf09f0061b2"). InnerVolumeSpecName "kube-api-access-gtgqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:37:49.912039 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.911951    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtgqg\" (UniqueName: \"kubernetes.io/projected/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-kube-api-access-gtgqg\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:37:49.912039 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.912001    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:37:49.912039 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.912012    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:37:49.912039 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:49.912020    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:37:50.335609 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:50.335569    2576 generic.go:356] "Generic (PLEG): container finished" podID="f4bb2f25-9952-4bef-8c2d-5bf09f0061b2" containerID="73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17" exitCode=0
Aug 04 18:37:50.336019 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:50.335621    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"
Aug 04 18:37:50.336019 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:50.335654    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd" event={"ID":"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2","Type":"ContainerDied","Data":"73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17"}
Aug 04 18:37:50.336019 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:50.335731    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd" event={"ID":"f4bb2f25-9952-4bef-8c2d-5bf09f0061b2","Type":"ContainerDied","Data":"2cac863c6b551899253c58ffa331c34b3f30e6e2f38f9f113b2730416a5a7994"}
Aug 04 18:37:50.336019 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:50.335758    2576 scope.go:117] "RemoveContainer" containerID="73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17"
Aug 04 18:37:50.345664 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:50.345641    2576 scope.go:117] "RemoveContainer" containerID="73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17"
Aug 04 18:37:50.345934 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:37:50.345914    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17\": container with ID starting with 73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17 not found: ID does not exist" containerID="73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17"
Aug 04 18:37:50.345989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:50.345944    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17"} err="failed to get container status \"73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17\": rpc error: code = NotFound desc = could not find container \"73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17\": container with ID starting with 73f4443de33df05831a4bb211a923d06a1fc7e14092c578b604cebf06b571f17 not found: ID does not exist"
Aug 04 18:37:50.362196 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:50.362166    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"]
Aug 04 18:37:50.367673 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:50.367645    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679694n6hd"]
Aug 04 18:37:50.582175 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:50.582143    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4bb2f25-9952-4bef-8c2d-5bf09f0061b2" path="/var/lib/kubelet/pods/f4bb2f25-9952-4bef-8c2d-5bf09f0061b2/volumes"
Aug 04 18:37:54.717127 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:37:54.717077    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d" podUID="cb70ec4d-80d5-4856-88f3-ffc8d420b628" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:38:03.714824 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:03.714788    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:03.852695 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:03.852645    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:03.947812 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:03.947771    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:04.139528 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:04.139488    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:04.342588 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:04.342544    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d" podUID="cb70ec4d-80d5-4856-88f3-ffc8d420b628" containerName="main" containerID="cri-o://e89dae5e74cbff87c96b2d0d10a767dab79b7cb65fe5a358ea32b6d54a69bde5" gracePeriod=45
Aug 04 18:38:04.606281 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:04.606250    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"
Aug 04 18:38:04.746878 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:04.746798    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb70ec4d-80d5-4856-88f3-ffc8d420b628-tls-certs\") pod \"cb70ec4d-80d5-4856-88f3-ffc8d420b628\" (UID: \"cb70ec4d-80d5-4856-88f3-ffc8d420b628\") "
Aug 04 18:38:04.747275 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:04.746890    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f96bz\" (UniqueName: \"kubernetes.io/projected/cb70ec4d-80d5-4856-88f3-ffc8d420b628-kube-api-access-f96bz\") pod \"cb70ec4d-80d5-4856-88f3-ffc8d420b628\" (UID: \"cb70ec4d-80d5-4856-88f3-ffc8d420b628\") "
Aug 04 18:38:04.748964 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:04.748935    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb70ec4d-80d5-4856-88f3-ffc8d420b628-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb70ec4d-80d5-4856-88f3-ffc8d420b628" (UID: "cb70ec4d-80d5-4856-88f3-ffc8d420b628"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:38:04.749028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:04.748962    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb70ec4d-80d5-4856-88f3-ffc8d420b628-kube-api-access-f96bz" (OuterVolumeSpecName: "kube-api-access-f96bz") pod "cb70ec4d-80d5-4856-88f3-ffc8d420b628" (UID: "cb70ec4d-80d5-4856-88f3-ffc8d420b628"). InnerVolumeSpecName "kube-api-access-f96bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:38:04.848729 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:04.848695    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb70ec4d-80d5-4856-88f3-ffc8d420b628-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:38:04.848729 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:04.848726    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f96bz\" (UniqueName: \"kubernetes.io/projected/cb70ec4d-80d5-4856-88f3-ffc8d420b628-kube-api-access-f96bz\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:38:05.408919 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:05.408886    2576 generic.go:356] "Generic (PLEG): container finished" podID="cb70ec4d-80d5-4856-88f3-ffc8d420b628" containerID="e89dae5e74cbff87c96b2d0d10a767dab79b7cb65fe5a358ea32b6d54a69bde5" exitCode=0
Aug 04 18:38:05.408919 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:05.408924    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d" event={"ID":"cb70ec4d-80d5-4856-88f3-ffc8d420b628","Type":"ContainerDied","Data":"e89dae5e74cbff87c96b2d0d10a767dab79b7cb65fe5a358ea32b6d54a69bde5"}
Aug 04 18:38:05.409269 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:05.408935    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"
Aug 04 18:38:05.409269 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:05.408965    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d" event={"ID":"cb70ec4d-80d5-4856-88f3-ffc8d420b628","Type":"ContainerDied","Data":"62e7a167e086d9822093ea85e1d0809b88705df9621dfdb7964187b7eac5cccc"}
Aug 04 18:38:05.409269 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:05.408986    2576 scope.go:117] "RemoveContainer" containerID="e89dae5e74cbff87c96b2d0d10a767dab79b7cb65fe5a358ea32b6d54a69bde5"
Aug 04 18:38:05.418300 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:05.418282    2576 scope.go:117] "RemoveContainer" containerID="e89dae5e74cbff87c96b2d0d10a767dab79b7cb65fe5a358ea32b6d54a69bde5"
Aug 04 18:38:05.418544 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:38:05.418522    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e89dae5e74cbff87c96b2d0d10a767dab79b7cb65fe5a358ea32b6d54a69bde5\": container with ID starting with e89dae5e74cbff87c96b2d0d10a767dab79b7cb65fe5a358ea32b6d54a69bde5 not found: ID does not exist" containerID="e89dae5e74cbff87c96b2d0d10a767dab79b7cb65fe5a358ea32b6d54a69bde5"
Aug 04 18:38:05.418609 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:05.418551    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89dae5e74cbff87c96b2d0d10a767dab79b7cb65fe5a358ea32b6d54a69bde5"} err="failed to get container status \"e89dae5e74cbff87c96b2d0d10a767dab79b7cb65fe5a358ea32b6d54a69bde5\": rpc error: code = NotFound desc = could not find container \"e89dae5e74cbff87c96b2d0d10a767dab79b7cb65fe5a358ea32b6d54a69bde5\": container with ID starting with e89dae5e74cbff87c96b2d0d10a767dab79b7cb65fe5a358ea32b6d54a69bde5 not found: ID does not exist"
Aug 04 18:38:05.431634 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:05.431609    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"]
Aug 04 18:38:05.437532 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:05.437509    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schez8f6d"]
Aug 04 18:38:06.582456 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:06.582427    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb70ec4d-80d5-4856-88f3-ffc8d420b628" path="/var/lib/kubelet/pods/cb70ec4d-80d5-4856-88f3-ffc8d420b628/volumes"
Aug 04 18:38:08.788398 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:08.788358    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:09.380213 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:09.380177    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:09.472998 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:09.472952    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:15.631311 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:15.631270    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:17.106646 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.106606    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"]
Aug 04 18:38:17.107161 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.106978    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb70ec4d-80d5-4856-88f3-ffc8d420b628" containerName="main"
Aug 04 18:38:17.107161 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.106991    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb70ec4d-80d5-4856-88f3-ffc8d420b628" containerName="main"
Aug 04 18:38:17.107161 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.107011    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4bb2f25-9952-4bef-8c2d-5bf09f0061b2" containerName="main"
Aug 04 18:38:17.107161 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.107018    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4bb2f25-9952-4bef-8c2d-5bf09f0061b2" containerName="main"
Aug 04 18:38:17.107161 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.107083    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb70ec4d-80d5-4856-88f3-ffc8d420b628" containerName="main"
Aug 04 18:38:17.107161 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.107095    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4bb2f25-9952-4bef-8c2d-5bf09f0061b2" containerName="main"
Aug 04 18:38:17.110269 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.110251    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.114393 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.114371    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Aug 04 18:38:17.114481 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.114440    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-gll56\""
Aug 04 18:38:17.116406 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.116389    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Aug 04 18:38:17.118209 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.118179    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Aug 04 18:38:17.127914 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.127893    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"]
Aug 04 18:38:17.140925 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.140890    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:17.160284 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.160248    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.160393 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.160303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-dshm\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.160393 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.160333    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxxct\" (UniqueName: \"kubernetes.io/projected/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-kube-api-access-vxxct\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.160393 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.160354    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.160514 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.160469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-home\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.160514 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.160504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.261711 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.261677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-dshm\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.261711 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.261715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxxct\" (UniqueName: \"kubernetes.io/projected/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-kube-api-access-vxxct\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.261970 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.261738    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.261970 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.261794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-home\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.261970 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.261814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.261970 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.261854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.262284 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.262260    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.262327 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.262278    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.262376 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.262321    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-home\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.264344 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.264320    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-dshm\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.264487 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.264466    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.271024 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.271003    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxxct\" (UniqueName: \"kubernetes.io/projected/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-kube-api-access-vxxct\") pod \"delw-v1-kserve-6cf8cbb847-88862\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.421914 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.421825    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:17.580694 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:17.580656    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"]
Aug 04 18:38:17.585510 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:38:17.585477    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64055cb1_13e2_4c6b_a54f_a00e82dd2a5c.slice/crio-567d008863f6d0b603746f0a087b222e547cb6f8ecb052db2c99091ce48d8285 WatchSource:0}: Error finding container 567d008863f6d0b603746f0a087b222e547cb6f8ecb052db2c99091ce48d8285: Status 404 returned error can't find the container with id 567d008863f6d0b603746f0a087b222e547cb6f8ecb052db2c99091ce48d8285
Aug 04 18:38:18.476165 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:18.476102    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862" event={"ID":"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c","Type":"ContainerStarted","Data":"b8ce978ab7ab8a79a6f51d42390452ad49f878464293e2b3b891f1682e0c19e6"}
Aug 04 18:38:18.476553 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:18.476184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862" event={"ID":"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c","Type":"ContainerStarted","Data":"567d008863f6d0b603746f0a087b222e547cb6f8ecb052db2c99091ce48d8285"}
Aug 04 18:38:18.495599 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:18.495547    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862" podStartSLOduration=1.4955316920000001 podStartE2EDuration="1.495531692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:38:18.493424051 +0000 UTC m=+2298.469255744" watchObservedRunningTime="2026-08-04 18:38:18.495531692 +0000 UTC m=+2298.471363389"
Aug 04 18:38:19.422944 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:19.422909    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:21.261758 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:21.261711    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:21.362724 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:21.362681    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:27.422445 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:27.422403    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:27.434879 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:27.434853    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:27.528217 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:27.528189    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:38:31.728860 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:31.728828    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:43.633482 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:43.633448    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:43.870320 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:43.870285    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"]
Aug 04 18:38:43.873822 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:43.873802    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"
Aug 04 18:38:43.877482 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:43.877457    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Aug 04 18:38:43.877632 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:43.877481    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-2s5gb\""
Aug 04 18:38:43.877632 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:43.877521    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Aug 04 18:38:43.877632 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:43.877459    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Aug 04 18:38:43.877632 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:43.877459    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-s6kmz\""
Aug 04 18:38:43.882644 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:43.882623    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"]
Aug 04 18:38:43.889503 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:43.889442    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:43.998698 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:43.998664    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30212333-84f8-43ae-a54d-1dc054f943c8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t\" (UID: \"30212333-84f8-43ae-a54d-1dc054f943c8\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"
Aug 04 18:38:43.998886 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:43.998711    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kznnc\" (UniqueName: \"kubernetes.io/projected/30212333-84f8-43ae-a54d-1dc054f943c8-kube-api-access-kznnc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t\" (UID: \"30212333-84f8-43ae-a54d-1dc054f943c8\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"
Aug 04 18:38:44.099557 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:44.099518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30212333-84f8-43ae-a54d-1dc054f943c8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t\" (UID: \"30212333-84f8-43ae-a54d-1dc054f943c8\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"
Aug 04 18:38:44.099733 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:44.099579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kznnc\" (UniqueName: \"kubernetes.io/projected/30212333-84f8-43ae-a54d-1dc054f943c8-kube-api-access-kznnc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t\" (UID: \"30212333-84f8-43ae-a54d-1dc054f943c8\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"
Aug 04 18:38:44.102123 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:44.102080    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30212333-84f8-43ae-a54d-1dc054f943c8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t\" (UID: \"30212333-84f8-43ae-a54d-1dc054f943c8\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"
Aug 04 18:38:44.108042 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:44.108018    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kznnc\" (UniqueName: \"kubernetes.io/projected/30212333-84f8-43ae-a54d-1dc054f943c8-kube-api-access-kznnc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t\" (UID: \"30212333-84f8-43ae-a54d-1dc054f943c8\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"
Aug 04 18:38:44.186183 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:44.186088    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"
Aug 04 18:38:44.320966 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:44.320932    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"]
Aug 04 18:38:44.324252 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:38:44.324221    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30212333_84f8_43ae_a54d_1dc054f943c8.slice/crio-937409a706aa92fda042c1d0d1dd8bcb6f83e14b0ccd0650165bdb294822b761 WatchSource:0}: Error finding container 937409a706aa92fda042c1d0d1dd8bcb6f83e14b0ccd0650165bdb294822b761: Status 404 returned error can't find the container with id 937409a706aa92fda042c1d0d1dd8bcb6f83e14b0ccd0650165bdb294822b761
Aug 04 18:38:44.609523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:44.609482    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t" event={"ID":"30212333-84f8-43ae-a54d-1dc054f943c8","Type":"ContainerStarted","Data":"1fef365f59861c3bd17f1979a651eed4270fda6a26ec2302a0509e6a739c937c"}
Aug 04 18:38:44.609732 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:44.609574    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t" event={"ID":"30212333-84f8-43ae-a54d-1dc054f943c8","Type":"ContainerStarted","Data":"937409a706aa92fda042c1d0d1dd8bcb6f83e14b0ccd0650165bdb294822b761"}
Aug 04 18:38:44.609818 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:44.609748    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"
Aug 04 18:38:44.627638 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:44.627586    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t" podStartSLOduration=1.627569882 podStartE2EDuration="1.627569882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:38:44.626611168 +0000 UTC m=+2324.602442863" watchObservedRunningTime="2026-08-04 18:38:44.627569882 +0000 UTC m=+2324.603401580"
Aug 04 18:38:46.365492 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:46.365457    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:51.934661 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:51.934627    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:52.036070 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:38:52.036031    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:02.789793 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:02.789758    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:02.889674 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:02.889629    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:07.024348 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:07.024305    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:15.615857 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:15.615823    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"
Aug 04 18:39:17.150469 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:17.150434    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:18.519929 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:18.519893    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:19.012371 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:19.012327    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:19.123289 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:19.123262    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:34.810797 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:34.810763    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:34.881432 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:34.881396    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"]
Aug 04 18:39:34.909688 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:34.909646    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:39.868673 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:39.868640    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:43.358675 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:43.358642    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:49.900323 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:49.900281    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862" podUID="64055cb1-13e2-4c6b-a54f-a00e82dd2a5c" containerName="main" containerID="cri-o://b8ce978ab7ab8a79a6f51d42390452ad49f878464293e2b3b891f1682e0c19e6" gracePeriod=45
Aug 04 18:39:50.148407 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.148378    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:39:50.190296 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.190218    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-dshm\") pod \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") "
Aug 04 18:39:50.190296 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.190251    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-home\") pod \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") "
Aug 04 18:39:50.191028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.190347    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-tls-certs\") pod \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") "
Aug 04 18:39:50.191028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.190405    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxxct\" (UniqueName: \"kubernetes.io/projected/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-kube-api-access-vxxct\") pod \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") "
Aug 04 18:39:50.191028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.190436    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-model-cache\") pod \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") "
Aug 04 18:39:50.191028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.190456    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-tmp-dir\") pod \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\" (UID: \"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c\") "
Aug 04 18:39:50.191028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.190539    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-home" (OuterVolumeSpecName: "home") pod "64055cb1-13e2-4c6b-a54f-a00e82dd2a5c" (UID: "64055cb1-13e2-4c6b-a54f-a00e82dd2a5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:39:50.191028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.190754    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-model-cache" (OuterVolumeSpecName: "model-cache") pod "64055cb1-13e2-4c6b-a54f-a00e82dd2a5c" (UID: "64055cb1-13e2-4c6b-a54f-a00e82dd2a5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:39:50.191028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.190938    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:39:50.191028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.190967    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:39:50.191028 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.190938    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64055cb1-13e2-4c6b-a54f-a00e82dd2a5c" (UID: "64055cb1-13e2-4c6b-a54f-a00e82dd2a5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:39:50.193014 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.192990    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-dshm" (OuterVolumeSpecName: "dshm") pod "64055cb1-13e2-4c6b-a54f-a00e82dd2a5c" (UID: "64055cb1-13e2-4c6b-a54f-a00e82dd2a5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:39:50.193131 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.193022    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-kube-api-access-vxxct" (OuterVolumeSpecName: "kube-api-access-vxxct") pod "64055cb1-13e2-4c6b-a54f-a00e82dd2a5c" (UID: "64055cb1-13e2-4c6b-a54f-a00e82dd2a5c"). InnerVolumeSpecName "kube-api-access-vxxct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:39:50.193287 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.193270    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64055cb1-13e2-4c6b-a54f-a00e82dd2a5c" (UID: "64055cb1-13e2-4c6b-a54f-a00e82dd2a5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:39:50.291862 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.291826    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:39:50.291862 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.291856    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxxct\" (UniqueName: \"kubernetes.io/projected/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-kube-api-access-vxxct\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:39:50.291862 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.291867    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:39:50.292107 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.291879    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:39:50.927918 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.927816    2576 generic.go:356] "Generic (PLEG): container finished" podID="64055cb1-13e2-4c6b-a54f-a00e82dd2a5c" containerID="b8ce978ab7ab8a79a6f51d42390452ad49f878464293e2b3b891f1682e0c19e6" exitCode=0
Aug 04 18:39:50.927918 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.927859    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862" event={"ID":"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c","Type":"ContainerDied","Data":"b8ce978ab7ab8a79a6f51d42390452ad49f878464293e2b3b891f1682e0c19e6"}
Aug 04 18:39:50.927918 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.927883    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"
Aug 04 18:39:50.927918 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.927899    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862" event={"ID":"64055cb1-13e2-4c6b-a54f-a00e82dd2a5c","Type":"ContainerDied","Data":"567d008863f6d0b603746f0a087b222e547cb6f8ecb052db2c99091ce48d8285"}
Aug 04 18:39:50.927918 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.927920    2576 scope.go:117] "RemoveContainer" containerID="b8ce978ab7ab8a79a6f51d42390452ad49f878464293e2b3b891f1682e0c19e6"
Aug 04 18:39:50.936587 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.936565    2576 scope.go:117] "RemoveContainer" containerID="b8ce978ab7ab8a79a6f51d42390452ad49f878464293e2b3b891f1682e0c19e6"
Aug 04 18:39:50.936869 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:39:50.936847    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ce978ab7ab8a79a6f51d42390452ad49f878464293e2b3b891f1682e0c19e6\": container with ID starting with b8ce978ab7ab8a79a6f51d42390452ad49f878464293e2b3b891f1682e0c19e6 not found: ID does not exist" containerID="b8ce978ab7ab8a79a6f51d42390452ad49f878464293e2b3b891f1682e0c19e6"
Aug 04 18:39:50.936924 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.936878    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ce978ab7ab8a79a6f51d42390452ad49f878464293e2b3b891f1682e0c19e6"} err="failed to get container status \"b8ce978ab7ab8a79a6f51d42390452ad49f878464293e2b3b891f1682e0c19e6\": rpc error: code = NotFound desc = could not find container \"b8ce978ab7ab8a79a6f51d42390452ad49f878464293e2b3b891f1682e0c19e6\": container with ID starting with b8ce978ab7ab8a79a6f51d42390452ad49f878464293e2b3b891f1682e0c19e6 not found: ID does not exist"
Aug 04 18:39:50.946631 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.946600    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"]
Aug 04 18:39:50.952993 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:50.952964    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-88862"]
Aug 04 18:39:52.582439 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:52.582406    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64055cb1-13e2-4c6b-a54f-a00e82dd2a5c" path="/var/lib/kubelet/pods/64055cb1-13e2-4c6b-a54f-a00e82dd2a5c/volumes"
Aug 04 18:39:58.594964 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.594929    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"]
Aug 04 18:39:58.595586 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.595564    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64055cb1-13e2-4c6b-a54f-a00e82dd2a5c" containerName="main"
Aug 04 18:39:58.595586 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.595587    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="64055cb1-13e2-4c6b-a54f-a00e82dd2a5c" containerName="main"
Aug 04 18:39:58.595751 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.595709    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="64055cb1-13e2-4c6b-a54f-a00e82dd2a5c" containerName="main"
Aug 04 18:39:58.600425 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.600405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.603387 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.603366    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-nr6j4\""
Aug 04 18:39:58.604410 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.604388    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Aug 04 18:39:58.604502 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.604394    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Aug 04 18:39:58.604502 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.604467    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Aug 04 18:39:58.611279 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.611047    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"]
Aug 04 18:39:58.626588 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.626554    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:58.675446 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.675409    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.675615 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.675472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554555ed-65e2-4a53-b120-509b27f17da0-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.675615 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.675524    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-home\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.675615 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.675544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.675615 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.675589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjh7\" (UniqueName: \"kubernetes.io/projected/554555ed-65e2-4a53-b120-509b27f17da0-kube-api-access-5wjh7\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.675755 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.675665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-dshm\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.776727 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.776683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554555ed-65e2-4a53-b120-509b27f17da0-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.776923 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.776736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-home\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.776923 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.776765    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.776923 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.776788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjh7\" (UniqueName: \"kubernetes.io/projected/554555ed-65e2-4a53-b120-509b27f17da0-kube-api-access-5wjh7\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.776923 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.776846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-dshm\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.776923 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.776892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.777261 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.777239    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-home\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.777323 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.777286    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.777368 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.777295    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.779049 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.779020    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-dshm\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.779442 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.779423    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554555ed-65e2-4a53-b120-509b27f17da0-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.785672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.785640    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjh7\" (UniqueName: \"kubernetes.io/projected/554555ed-65e2-4a53-b120-509b27f17da0-kube-api-access-5wjh7\") pod \"roll-v2-kserve-d4dbf9d67-fnn4l\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:58.912527 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:58.912437    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:39:59.047134 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:59.047089    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"]
Aug 04 18:39:59.049224 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:39:59.049191    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554555ed_65e2_4a53_b120_509b27f17da0.slice/crio-d8200a9e1a23ffa4866234bdb7721955093aeee78b271e6fb825929ed16835ed WatchSource:0}: Error finding container d8200a9e1a23ffa4866234bdb7721955093aeee78b271e6fb825929ed16835ed: Status 404 returned error can't find the container with id d8200a9e1a23ffa4866234bdb7721955093aeee78b271e6fb825929ed16835ed
Aug 04 18:39:59.976989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:59.976953    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l" event={"ID":"554555ed-65e2-4a53-b120-509b27f17da0","Type":"ContainerStarted","Data":"721df07e3591e973479fb415eaa815ed6a10039ba266ab9b279ac9c5c2be9571"}
Aug 04 18:39:59.977384 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:59.977006    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l" event={"ID":"554555ed-65e2-4a53-b120-509b27f17da0","Type":"ContainerStarted","Data":"d8200a9e1a23ffa4866234bdb7721955093aeee78b271e6fb825929ed16835ed"}
Aug 04 18:39:59.999627 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:39:59.999575    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l" podStartSLOduration=1.9995598270000001 podStartE2EDuration="1.999559827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:39:59.997364531 +0000 UTC m=+2399.973196232" watchObservedRunningTime="2026-08-04 18:39:59.999559827 +0000 UTC m=+2399.975391524"
Aug 04 18:40:00.912877 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:00.912837    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:40:08.913217 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:08.913180    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:40:08.925877 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:08.925843    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:40:09.030572 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:09.030542    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:40:17.093017 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:17.092968    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:40:22.659060 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:22.658973    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:40:22.760047 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:22.760014    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:40:36.683321 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:36.683284    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"]
Aug 04 18:40:36.700391 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:36.700347    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:40:36.800939 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:36.800898    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:40:45.616036 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:45.615993    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t" podUID="30212333-84f8-43ae-a54d-1dc054f943c8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:40:51.702987 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:51.702916    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t" podUID="30212333-84f8-43ae-a54d-1dc054f943c8" containerName="main" containerID="cri-o://1fef365f59861c3bd17f1979a651eed4270fda6a26ec2302a0509e6a739c937c" gracePeriod=45
Aug 04 18:40:51.951138 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:51.951094    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"
Aug 04 18:40:52.056661 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.056634    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kznnc\" (UniqueName: \"kubernetes.io/projected/30212333-84f8-43ae-a54d-1dc054f943c8-kube-api-access-kznnc\") pod \"30212333-84f8-43ae-a54d-1dc054f943c8\" (UID: \"30212333-84f8-43ae-a54d-1dc054f943c8\") "
Aug 04 18:40:52.056849 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.056693    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30212333-84f8-43ae-a54d-1dc054f943c8-tls-certs\") pod \"30212333-84f8-43ae-a54d-1dc054f943c8\" (UID: \"30212333-84f8-43ae-a54d-1dc054f943c8\") "
Aug 04 18:40:52.058809 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.058782    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30212333-84f8-43ae-a54d-1dc054f943c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30212333-84f8-43ae-a54d-1dc054f943c8" (UID: "30212333-84f8-43ae-a54d-1dc054f943c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:40:52.058924 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.058902    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30212333-84f8-43ae-a54d-1dc054f943c8-kube-api-access-kznnc" (OuterVolumeSpecName: "kube-api-access-kznnc") pod "30212333-84f8-43ae-a54d-1dc054f943c8" (UID: "30212333-84f8-43ae-a54d-1dc054f943c8"). InnerVolumeSpecName "kube-api-access-kznnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:40:52.157551 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.157511    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kznnc\" (UniqueName: \"kubernetes.io/projected/30212333-84f8-43ae-a54d-1dc054f943c8-kube-api-access-kznnc\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:40:52.157551 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.157545    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30212333-84f8-43ae-a54d-1dc054f943c8-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:40:52.233861 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.233825    2576 generic.go:356] "Generic (PLEG): container finished" podID="30212333-84f8-43ae-a54d-1dc054f943c8" containerID="1fef365f59861c3bd17f1979a651eed4270fda6a26ec2302a0509e6a739c937c" exitCode=0
Aug 04 18:40:52.234046 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.233874    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"
Aug 04 18:40:52.234046 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.233918    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t" event={"ID":"30212333-84f8-43ae-a54d-1dc054f943c8","Type":"ContainerDied","Data":"1fef365f59861c3bd17f1979a651eed4270fda6a26ec2302a0509e6a739c937c"}
Aug 04 18:40:52.234046 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.233975    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t" event={"ID":"30212333-84f8-43ae-a54d-1dc054f943c8","Type":"ContainerDied","Data":"937409a706aa92fda042c1d0d1dd8bcb6f83e14b0ccd0650165bdb294822b761"}
Aug 04 18:40:52.234046 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.234005    2576 scope.go:117] "RemoveContainer" containerID="1fef365f59861c3bd17f1979a651eed4270fda6a26ec2302a0509e6a739c937c"
Aug 04 18:40:52.242758 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.242737    2576 scope.go:117] "RemoveContainer" containerID="1fef365f59861c3bd17f1979a651eed4270fda6a26ec2302a0509e6a739c937c"
Aug 04 18:40:52.243011 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:40:52.242992    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fef365f59861c3bd17f1979a651eed4270fda6a26ec2302a0509e6a739c937c\": container with ID starting with 1fef365f59861c3bd17f1979a651eed4270fda6a26ec2302a0509e6a739c937c not found: ID does not exist" containerID="1fef365f59861c3bd17f1979a651eed4270fda6a26ec2302a0509e6a739c937c"
Aug 04 18:40:52.243057 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.243020    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fef365f59861c3bd17f1979a651eed4270fda6a26ec2302a0509e6a739c937c"} err="failed to get container status \"1fef365f59861c3bd17f1979a651eed4270fda6a26ec2302a0509e6a739c937c\": rpc error: code = NotFound desc = could not find container \"1fef365f59861c3bd17f1979a651eed4270fda6a26ec2302a0509e6a739c937c\": container with ID starting with 1fef365f59861c3bd17f1979a651eed4270fda6a26ec2302a0509e6a739c937c not found: ID does not exist"
Aug 04 18:40:52.255665 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.255636    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"]
Aug 04 18:40:52.261464 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.261438    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewjd7t"]
Aug 04 18:40:52.582830 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:52.582798    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30212333-84f8-43ae-a54d-1dc054f943c8" path="/var/lib/kubelet/pods/30212333-84f8-43ae-a54d-1dc054f943c8/volumes"
Aug 04 18:40:56.737775 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:56.737735    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:40:56.833264 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:56.833226    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:40:59.156157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:59.156090    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:40:59.292524 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:59.292493    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:40:59.385293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:40:59.385257    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:01.605014 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.604315    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:01.702082 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.702044    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:01.837515 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.837479    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"]
Aug 04 18:41:01.837889 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.837876    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30212333-84f8-43ae-a54d-1dc054f943c8" containerName="main"
Aug 04 18:41:01.837942 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.837890    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="30212333-84f8-43ae-a54d-1dc054f943c8" containerName="main"
Aug 04 18:41:01.837981 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.837948    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="30212333-84f8-43ae-a54d-1dc054f943c8" containerName="main"
Aug 04 18:41:01.842297 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.842270    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"
Aug 04 18:41:01.845959 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.845928    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-wjssv\""
Aug 04 18:41:01.845959 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.845952    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Aug 04 18:41:01.846192 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.845991    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Aug 04 18:41:01.846192 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.845933    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Aug 04 18:41:01.846192 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.846090    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-m8672\""
Aug 04 18:41:01.849069 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.849045    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"]
Aug 04 18:41:01.911086 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.910988    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:01.947574 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.947537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gllq\" (UniqueName: \"kubernetes.io/projected/330a5cd8-6657-4e5f-bfac-46eace850f4f-kube-api-access-7gllq\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl\" (UID: \"330a5cd8-6657-4e5f-bfac-46eace850f4f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"
Aug 04 18:41:01.947730 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:01.947677    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/330a5cd8-6657-4e5f-bfac-46eace850f4f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl\" (UID: \"330a5cd8-6657-4e5f-bfac-46eace850f4f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"
Aug 04 18:41:02.048755 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:02.048716    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/330a5cd8-6657-4e5f-bfac-46eace850f4f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl\" (UID: \"330a5cd8-6657-4e5f-bfac-46eace850f4f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"
Aug 04 18:41:02.048916 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:02.048858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gllq\" (UniqueName: \"kubernetes.io/projected/330a5cd8-6657-4e5f-bfac-46eace850f4f-kube-api-access-7gllq\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl\" (UID: \"330a5cd8-6657-4e5f-bfac-46eace850f4f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"
Aug 04 18:41:02.051308 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:02.051286    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/330a5cd8-6657-4e5f-bfac-46eace850f4f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl\" (UID: \"330a5cd8-6657-4e5f-bfac-46eace850f4f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"
Aug 04 18:41:02.058848 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:02.058818    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gllq\" (UniqueName: \"kubernetes.io/projected/330a5cd8-6657-4e5f-bfac-46eace850f4f-kube-api-access-7gllq\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl\" (UID: \"330a5cd8-6657-4e5f-bfac-46eace850f4f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"
Aug 04 18:41:02.154985 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:02.154945    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"
Aug 04 18:41:02.290196 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:02.290125    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"]
Aug 04 18:41:02.292386 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:41:02.292356    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330a5cd8_6657_4e5f_bfac_46eace850f4f.slice/crio-a61f9b76bd46309c91a3a1df3e4a9211c7c2b74e0c6994a86a8c9cb969f19dbb WatchSource:0}: Error finding container a61f9b76bd46309c91a3a1df3e4a9211c7c2b74e0c6994a86a8c9cb969f19dbb: Status 404 returned error can't find the container with id a61f9b76bd46309c91a3a1df3e4a9211c7c2b74e0c6994a86a8c9cb969f19dbb
Aug 04 18:41:03.298312 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:03.298272    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl" event={"ID":"330a5cd8-6657-4e5f-bfac-46eace850f4f","Type":"ContainerStarted","Data":"d1365e888c33fac571307f7e32e69d2571aa68ee697e18896aa98d863a5bb881"}
Aug 04 18:41:03.298727 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:03.298336    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl" event={"ID":"330a5cd8-6657-4e5f-bfac-46eace850f4f","Type":"ContainerStarted","Data":"a61f9b76bd46309c91a3a1df3e4a9211c7c2b74e0c6994a86a8c9cb969f19dbb"}
Aug 04 18:41:03.298727 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:03.298368    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"
Aug 04 18:41:03.316690 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:03.316630    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl" podStartSLOduration=2.316609752 podStartE2EDuration="2.316609752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:41:03.315543 +0000 UTC m=+2463.291374696" watchObservedRunningTime="2026-08-04 18:41:03.316609752 +0000 UTC m=+2463.292441449"
Aug 04 18:41:04.000617 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:04.000580    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:04.137180 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:04.137147    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"]
Aug 04 18:41:04.227011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:04.226976    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:05.108413 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:05.108365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:05.627199 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:05.627163    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:09.811624 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:09.811582    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:10.571064 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:10.571024    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:19.156912 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.156867    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l" podUID="554555ed-65e2-4a53-b120-509b27f17da0" containerName="main" containerID="cri-o://721df07e3591e973479fb415eaa815ed6a10039ba266ab9b279ac9c5c2be9571" gracePeriod=45
Aug 04 18:41:19.388277 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.388245    2576 generic.go:356] "Generic (PLEG): container finished" podID="554555ed-65e2-4a53-b120-509b27f17da0" containerID="721df07e3591e973479fb415eaa815ed6a10039ba266ab9b279ac9c5c2be9571" exitCode=0
Aug 04 18:41:19.388430 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.388319    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l" event={"ID":"554555ed-65e2-4a53-b120-509b27f17da0","Type":"ContainerDied","Data":"721df07e3591e973479fb415eaa815ed6a10039ba266ab9b279ac9c5c2be9571"}
Aug 04 18:41:19.404226 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.404201    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:41:19.514880 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.514792    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wjh7\" (UniqueName: \"kubernetes.io/projected/554555ed-65e2-4a53-b120-509b27f17da0-kube-api-access-5wjh7\") pod \"554555ed-65e2-4a53-b120-509b27f17da0\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") "
Aug 04 18:41:19.514880 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.514838    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-home\") pod \"554555ed-65e2-4a53-b120-509b27f17da0\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") "
Aug 04 18:41:19.514880 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.514881    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-tmp-dir\") pod \"554555ed-65e2-4a53-b120-509b27f17da0\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") "
Aug 04 18:41:19.515207 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.514906    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-model-cache\") pod \"554555ed-65e2-4a53-b120-509b27f17da0\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") "
Aug 04 18:41:19.515207 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.514953    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554555ed-65e2-4a53-b120-509b27f17da0-tls-certs\") pod \"554555ed-65e2-4a53-b120-509b27f17da0\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") "
Aug 04 18:41:19.515207 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.514995    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-dshm\") pod \"554555ed-65e2-4a53-b120-509b27f17da0\" (UID: \"554555ed-65e2-4a53-b120-509b27f17da0\") "
Aug 04 18:41:19.515207 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.515160    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-home" (OuterVolumeSpecName: "home") pod "554555ed-65e2-4a53-b120-509b27f17da0" (UID: "554555ed-65e2-4a53-b120-509b27f17da0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:41:19.515207 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.515195    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-model-cache" (OuterVolumeSpecName: "model-cache") pod "554555ed-65e2-4a53-b120-509b27f17da0" (UID: "554555ed-65e2-4a53-b120-509b27f17da0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:41:19.515399 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.515210    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "554555ed-65e2-4a53-b120-509b27f17da0" (UID: "554555ed-65e2-4a53-b120-509b27f17da0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:41:19.515399 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.515276    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:41:19.515399 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.515297    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:41:19.517197 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.517174    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/554555ed-65e2-4a53-b120-509b27f17da0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "554555ed-65e2-4a53-b120-509b27f17da0" (UID: "554555ed-65e2-4a53-b120-509b27f17da0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:41:19.517288 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.517241    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554555ed-65e2-4a53-b120-509b27f17da0-kube-api-access-5wjh7" (OuterVolumeSpecName: "kube-api-access-5wjh7") pod "554555ed-65e2-4a53-b120-509b27f17da0" (UID: "554555ed-65e2-4a53-b120-509b27f17da0"). InnerVolumeSpecName "kube-api-access-5wjh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:41:19.517288 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.517246    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-dshm" (OuterVolumeSpecName: "dshm") pod "554555ed-65e2-4a53-b120-509b27f17da0" (UID: "554555ed-65e2-4a53-b120-509b27f17da0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:41:19.616026 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.615982    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:41:19.616026 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.616020    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554555ed-65e2-4a53-b120-509b27f17da0-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:41:19.616026 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.616029    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554555ed-65e2-4a53-b120-509b27f17da0-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:41:19.616317 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:19.616039    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wjh7\" (UniqueName: \"kubernetes.io/projected/554555ed-65e2-4a53-b120-509b27f17da0-kube-api-access-5wjh7\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:41:20.398651 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:20.398616    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l" event={"ID":"554555ed-65e2-4a53-b120-509b27f17da0","Type":"ContainerDied","Data":"d8200a9e1a23ffa4866234bdb7721955093aeee78b271e6fb825929ed16835ed"}
Aug 04 18:41:20.399130 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:20.398673    2576 scope.go:117] "RemoveContainer" containerID="721df07e3591e973479fb415eaa815ed6a10039ba266ab9b279ac9c5c2be9571"
Aug 04 18:41:20.399130 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:20.398631    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"
Aug 04 18:41:20.422001 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:20.421972    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"]
Aug 04 18:41:20.427935 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:20.427894    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fnn4l"]
Aug 04 18:41:20.582095 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:20.582052    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="554555ed-65e2-4a53-b120-509b27f17da0" path="/var/lib/kubelet/pods/554555ed-65e2-4a53-b120-509b27f17da0/volumes"
Aug 04 18:41:25.140322 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:25.140288    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:34.304754 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:34.304724    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"
Aug 04 18:41:36.050720 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.050686    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"]
Aug 04 18:41:36.051176 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.051073    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="554555ed-65e2-4a53-b120-509b27f17da0" containerName="main"
Aug 04 18:41:36.051176 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.051084    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="554555ed-65e2-4a53-b120-509b27f17da0" containerName="main"
Aug 04 18:41:36.051273 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.051200    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="554555ed-65e2-4a53-b120-509b27f17da0" containerName="main"
Aug 04 18:41:36.055520 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.055500    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.072619 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.072593    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Aug 04 18:41:36.076008 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.075988    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Aug 04 18:41:36.076451 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.076436    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-pmzcb\""
Aug 04 18:41:36.076734 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.076710    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Aug 04 18:41:36.096672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.096638    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:36.096957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.096934    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"]
Aug 04 18:41:36.167172 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.167129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.167393 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.167181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02bbacfc-d045-4fe6-8858-11a734763a1c-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.167393 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.167321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-dshm\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.167393 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.167357    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-model-cache\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.167546 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.167395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhgfz\" (UniqueName: \"kubernetes.io/projected/02bbacfc-d045-4fe6-8858-11a734763a1c-kube-api-access-hhgfz\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.167546 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.167423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-home\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.268752 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.268705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-dshm\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.268752 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.268749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-model-cache\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.269015 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.268787    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhgfz\" (UniqueName: \"kubernetes.io/projected/02bbacfc-d045-4fe6-8858-11a734763a1c-kube-api-access-hhgfz\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.269015 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.268816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-home\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.269015 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.268875    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.269015 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.268911    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02bbacfc-d045-4fe6-8858-11a734763a1c-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.269292 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.269267    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-home\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.269350 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.269304    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-model-cache\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.269350 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.269339    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.271243 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.271220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-dshm\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.271390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.271375    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02bbacfc-d045-4fe6-8858-11a734763a1c-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.277795 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.277766    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhgfz\" (UniqueName: \"kubernetes.io/projected/02bbacfc-d045-4fe6-8858-11a734763a1c-kube-api-access-hhgfz\") pod \"owner-v2-kserve-5cf8959f95-65j4m\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.365409 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.365313    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:36.502156 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:36.502130    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"]
Aug 04 18:41:36.504977 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:41:36.504946    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bbacfc_d045_4fe6_8858_11a734763a1c.slice/crio-153ce39084bfbf6c4a9c91afc745f45bf9d8ad58b9dbdd025f37c44bade4f1df WatchSource:0}: Error finding container 153ce39084bfbf6c4a9c91afc745f45bf9d8ad58b9dbdd025f37c44bade4f1df: Status 404 returned error can't find the container with id 153ce39084bfbf6c4a9c91afc745f45bf9d8ad58b9dbdd025f37c44bade4f1df
Aug 04 18:41:37.483539 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:37.483500    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m" event={"ID":"02bbacfc-d045-4fe6-8858-11a734763a1c","Type":"ContainerStarted","Data":"6bfb0977f4cc62111784c2cd427b8c87857fbd1525849f27e0354c9605a4e632"}
Aug 04 18:41:37.483978 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:37.483556    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m" event={"ID":"02bbacfc-d045-4fe6-8858-11a734763a1c","Type":"ContainerStarted","Data":"153ce39084bfbf6c4a9c91afc745f45bf9d8ad58b9dbdd025f37c44bade4f1df"}
Aug 04 18:41:37.503485 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:37.503436    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m" podStartSLOduration=1.503420444 podStartE2EDuration="1.503420444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:41:37.502051526 +0000 UTC m=+2497.477883220" watchObservedRunningTime="2026-08-04 18:41:37.503420444 +0000 UTC m=+2497.479252141"
Aug 04 18:41:38.365852 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:38.365816    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:40.739724 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:40.739687    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:41.548060 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:41.548025    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:46.366052 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:46.366015    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:46.378891 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:46.378866    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:46.537633 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:46.537598    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:41:47.804522 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:47.804483    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:47.942958 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:41:47.942923    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:26.170484 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:26.170433    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"]
Aug 04 18:42:26.179352 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:26.179313    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:29.903971 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:29.903932    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:34.304914 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:34.304868    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl" podUID="330a5cd8-6657-4e5f-bfac-46eace850f4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:42:35.343964 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:35.343928    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:41.192095 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.192049    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl" podUID="330a5cd8-6657-4e5f-bfac-46eace850f4f" containerName="main" containerID="cri-o://d1365e888c33fac571307f7e32e69d2571aa68ee697e18896aa98d863a5bb881" gracePeriod=45
Aug 04 18:42:41.447765 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.447691    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"
Aug 04 18:42:41.562378 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.562342    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gllq\" (UniqueName: \"kubernetes.io/projected/330a5cd8-6657-4e5f-bfac-46eace850f4f-kube-api-access-7gllq\") pod \"330a5cd8-6657-4e5f-bfac-46eace850f4f\" (UID: \"330a5cd8-6657-4e5f-bfac-46eace850f4f\") "
Aug 04 18:42:41.562552 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.562400    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/330a5cd8-6657-4e5f-bfac-46eace850f4f-tls-certs\") pod \"330a5cd8-6657-4e5f-bfac-46eace850f4f\" (UID: \"330a5cd8-6657-4e5f-bfac-46eace850f4f\") "
Aug 04 18:42:41.564580 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.564553    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330a5cd8-6657-4e5f-bfac-46eace850f4f-kube-api-access-7gllq" (OuterVolumeSpecName: "kube-api-access-7gllq") pod "330a5cd8-6657-4e5f-bfac-46eace850f4f" (UID: "330a5cd8-6657-4e5f-bfac-46eace850f4f"). InnerVolumeSpecName "kube-api-access-7gllq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:42:41.564663 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.564584    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/330a5cd8-6657-4e5f-bfac-46eace850f4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "330a5cd8-6657-4e5f-bfac-46eace850f4f" (UID: "330a5cd8-6657-4e5f-bfac-46eace850f4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:42:41.663994 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.663960    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/330a5cd8-6657-4e5f-bfac-46eace850f4f-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:42:41.663994 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.663991    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gllq\" (UniqueName: \"kubernetes.io/projected/330a5cd8-6657-4e5f-bfac-46eace850f4f-kube-api-access-7gllq\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:42:41.797516 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.797480    2576 generic.go:356] "Generic (PLEG): container finished" podID="330a5cd8-6657-4e5f-bfac-46eace850f4f" containerID="d1365e888c33fac571307f7e32e69d2571aa68ee697e18896aa98d863a5bb881" exitCode=0
Aug 04 18:42:41.797694 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.797521    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"
Aug 04 18:42:41.797694 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.797524    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl" event={"ID":"330a5cd8-6657-4e5f-bfac-46eace850f4f","Type":"ContainerDied","Data":"d1365e888c33fac571307f7e32e69d2571aa68ee697e18896aa98d863a5bb881"}
Aug 04 18:42:41.797694 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.797659    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl" event={"ID":"330a5cd8-6657-4e5f-bfac-46eace850f4f","Type":"ContainerDied","Data":"a61f9b76bd46309c91a3a1df3e4a9211c7c2b74e0c6994a86a8c9cb969f19dbb"}
Aug 04 18:42:41.797694 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.797690    2576 scope.go:117] "RemoveContainer" containerID="d1365e888c33fac571307f7e32e69d2571aa68ee697e18896aa98d863a5bb881"
Aug 04 18:42:41.813341 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.813308    2576 scope.go:117] "RemoveContainer" containerID="d1365e888c33fac571307f7e32e69d2571aa68ee697e18896aa98d863a5bb881"
Aug 04 18:42:41.813617 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:42:41.813598    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1365e888c33fac571307f7e32e69d2571aa68ee697e18896aa98d863a5bb881\": container with ID starting with d1365e888c33fac571307f7e32e69d2571aa68ee697e18896aa98d863a5bb881 not found: ID does not exist" containerID="d1365e888c33fac571307f7e32e69d2571aa68ee697e18896aa98d863a5bb881"
Aug 04 18:42:41.813674 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.813625    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1365e888c33fac571307f7e32e69d2571aa68ee697e18896aa98d863a5bb881"} err="failed to get container status \"d1365e888c33fac571307f7e32e69d2571aa68ee697e18896aa98d863a5bb881\": rpc error: code = NotFound desc = could not find container \"d1365e888c33fac571307f7e32e69d2571aa68ee697e18896aa98d863a5bb881\": container with ID starting with d1365e888c33fac571307f7e32e69d2571aa68ee697e18896aa98d863a5bb881 not found: ID does not exist"
Aug 04 18:42:41.823875 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.823847    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"]
Aug 04 18:42:41.829602 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:41.829578    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche8hlcl"]
Aug 04 18:42:42.582548 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:42.582517    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="330a5cd8-6657-4e5f-bfac-46eace850f4f" path="/var/lib/kubelet/pods/330a5cd8-6657-4e5f-bfac-46eace850f4f/volumes"
Aug 04 18:42:45.653779 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:45.653745    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:45.752612 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:45.752578    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:47.884383 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:47.884347    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:47.991048 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:47.991015    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:48.093059 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:48.093027    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:52.525932 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:52.525897    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:53.777378 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:53.777340    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:53.850838 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:53.850804    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:57.020146 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.020095    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:57.115228 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.115192    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:57.251076 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.250915    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"]
Aug 04 18:42:57.251492 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.251390    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="330a5cd8-6657-4e5f-bfac-46eace850f4f" containerName="main"
Aug 04 18:42:57.251623 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.251514    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="330a5cd8-6657-4e5f-bfac-46eace850f4f" containerName="main"
Aug 04 18:42:57.251706 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.251690    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="330a5cd8-6657-4e5f-bfac-46eace850f4f" containerName="main"
Aug 04 18:42:57.256583 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.256554    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.262232 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.262204    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Aug 04 18:42:57.262443 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.262416    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Aug 04 18:42:57.262608 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.262524    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-87z96\""
Aug 04 18:42:57.266506 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.266478    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Aug 04 18:42:57.279648 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.279572    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"]
Aug 04 18:42:57.295280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.295241    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:57.297864 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.297827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.298040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.297903    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-token\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.298040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.297928    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74jk7\" (UniqueName: \"kubernetes.io/projected/57b0d619-7e8d-406b-aa0f-923a6e75f33a-kube-api-access-74jk7\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.298040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.297960    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-credential-socket\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.298040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.297997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-data\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.298040 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.298025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-workload-socket\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.298357 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.298063    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-envoy\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.298357 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.298148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-workload-certs\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.298357 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.298178    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-podinfo\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.396550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.396517    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:57.398611 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.398584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-workload-certs\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.398740 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.398625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-podinfo\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.398740 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.398665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.398740 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.398712    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-token\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.398902 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.398835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74jk7\" (UniqueName: \"kubernetes.io/projected/57b0d619-7e8d-406b-aa0f-923a6e75f33a-kube-api-access-74jk7\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.398902 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.398884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-credential-socket\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.399006 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.398928    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-data\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.399006 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.398960    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-workload-socket\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.399133 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.399028    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-workload-certs\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.399133 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.399073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-envoy\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.399321 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.399298    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-data\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.399481 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.399456    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-credential-socket\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.399670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.399478    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.399761 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.399521    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-workload-socket\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.401239 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.401220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-envoy\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.401482 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.401462    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-podinfo\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.422190 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.422165    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-token\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.422326 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.422167    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74jk7\" (UniqueName: \"kubernetes.io/projected/57b0d619-7e8d-406b-aa0f-923a6e75f33a-kube-api-access-74jk7\") pod \"router-gateway-1-openshift-default-77cc9b855f-s9pmd\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.568688 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.568658    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:57.718846 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.718820    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"]
Aug 04 18:42:57.721309 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:42:57.721269    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b0d619_7e8d_406b_aa0f_923a6e75f33a.slice/crio-0d3155681ec1f832d484d029fb72e3695b5fcfdd8501a1861b312638445b1168 WatchSource:0}: Error finding container 0d3155681ec1f832d484d029fb72e3695b5fcfdd8501a1861b312638445b1168: Status 404 returned error can't find the container with id 0d3155681ec1f832d484d029fb72e3695b5fcfdd8501a1861b312638445b1168
Aug 04 18:42:57.723440 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.723403    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:42:57.723535 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.723485    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:42:57.723535 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.723527    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:42:57.876860 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.876758    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd" event={"ID":"57b0d619-7e8d-406b-aa0f-923a6e75f33a","Type":"ContainerStarted","Data":"2a4d86b0906691e2784674820903cee3b1f7902ca56a02739998b28fb37a3dc3"}
Aug 04 18:42:57.876860 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.876826    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd" event={"ID":"57b0d619-7e8d-406b-aa0f-923a6e75f33a","Type":"ContainerStarted","Data":"0d3155681ec1f832d484d029fb72e3695b5fcfdd8501a1861b312638445b1168"}
Aug 04 18:42:57.901166 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:57.901090    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd" podStartSLOduration=0.901075611 podStartE2EDuration="901.075611ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:42:57.899567629 +0000 UTC m=+2577.875399338" watchObservedRunningTime="2026-08-04 18:42:57.901075611 +0000 UTC m=+2577.876907307"
Aug 04 18:42:58.569787 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:58.569749    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:58.574397 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:58.574368    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:58.603087 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:58.603055    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:58.881507 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:58.881417    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:58.882464 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:58.882441    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:42:59.015658 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:42:59.015609    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:01.287622 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:01.287221    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:02.099834 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:02.099798    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"]
Aug 04 18:43:02.161375 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:02.161344    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:12.199246 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.199209    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:12.305402 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.305369    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"]
Aug 04 18:43:12.308813 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.308790    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"
Aug 04 18:43:12.311616 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.311596    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Aug 04 18:43:12.312690 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.312666    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-pbh5l\""
Aug 04 18:43:12.312690 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.312679    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Aug 04 18:43:12.312870 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.312703    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-5b8h5\""
Aug 04 18:43:12.312870 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.312687    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Aug 04 18:43:12.318730 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.318704    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"]
Aug 04 18:43:12.325552 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.325527    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:12.448344 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.448309    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6zcw\" (UniqueName: \"kubernetes.io/projected/20b6a547-3ea8-4cf6-8723-2391b3989fb4-kube-api-access-d6zcw\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb\" (UID: \"20b6a547-3ea8-4cf6-8723-2391b3989fb4\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"
Aug 04 18:43:12.448499 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.448394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b6a547-3ea8-4cf6-8723-2391b3989fb4-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb\" (UID: \"20b6a547-3ea8-4cf6-8723-2391b3989fb4\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"
Aug 04 18:43:12.549157 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.549122    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b6a547-3ea8-4cf6-8723-2391b3989fb4-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb\" (UID: \"20b6a547-3ea8-4cf6-8723-2391b3989fb4\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"
Aug 04 18:43:12.549346 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.549219    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zcw\" (UniqueName: \"kubernetes.io/projected/20b6a547-3ea8-4cf6-8723-2391b3989fb4-kube-api-access-d6zcw\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb\" (UID: \"20b6a547-3ea8-4cf6-8723-2391b3989fb4\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"
Aug 04 18:43:12.551765 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.551744    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b6a547-3ea8-4cf6-8723-2391b3989fb4-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb\" (UID: \"20b6a547-3ea8-4cf6-8723-2391b3989fb4\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"
Aug 04 18:43:12.557600 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.557572    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6zcw\" (UniqueName: \"kubernetes.io/projected/20b6a547-3ea8-4cf6-8723-2391b3989fb4-kube-api-access-d6zcw\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb\" (UID: \"20b6a547-3ea8-4cf6-8723-2391b3989fb4\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"
Aug 04 18:43:12.620341 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.620308    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"
Aug 04 18:43:12.765892 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.765858    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"]
Aug 04 18:43:12.767977 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:43:12.767952    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b6a547_3ea8_4cf6_8723_2391b3989fb4.slice/crio-9804d9e4241fa5d91a65d72ce0c6565e40aea2aebc2fe583029139ce1b3837d6 WatchSource:0}: Error finding container 9804d9e4241fa5d91a65d72ce0c6565e40aea2aebc2fe583029139ce1b3837d6: Status 404 returned error can't find the container with id 9804d9e4241fa5d91a65d72ce0c6565e40aea2aebc2fe583029139ce1b3837d6
Aug 04 18:43:12.949090 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.949053    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb" event={"ID":"20b6a547-3ea8-4cf6-8723-2391b3989fb4","Type":"ContainerStarted","Data":"e8bc772ad4803938c30c6cbcd7ad5c6099229a4348c53395a33d0780f10540d3"}
Aug 04 18:43:12.949313 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.949125    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb" event={"ID":"20b6a547-3ea8-4cf6-8723-2391b3989fb4","Type":"ContainerStarted","Data":"9804d9e4241fa5d91a65d72ce0c6565e40aea2aebc2fe583029139ce1b3837d6"}
Aug 04 18:43:12.949313 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.949194    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"
Aug 04 18:43:12.967976 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:12.967915    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb" podStartSLOduration=0.967899243 podStartE2EDuration="967.899243ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:43:12.966245574 +0000 UTC m=+2592.942077274" watchObservedRunningTime="2026-08-04 18:43:12.967899243 +0000 UTC m=+2592.943730951"
Aug 04 18:43:14.595442 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:14.595396    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:17.119406 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.119357    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m" podUID="02bbacfc-d045-4fe6-8858-11a734763a1c" containerName="main" containerID="cri-o://6bfb0977f4cc62111784c2cd427b8c87857fbd1525849f27e0354c9605a4e632" gracePeriod=45
Aug 04 18:43:17.369309 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.369283    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:43:17.495432 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.495322    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-home\") pod \"02bbacfc-d045-4fe6-8858-11a734763a1c\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") "
Aug 04 18:43:17.495432 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.495376    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02bbacfc-d045-4fe6-8858-11a734763a1c-tls-certs\") pod \"02bbacfc-d045-4fe6-8858-11a734763a1c\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") "
Aug 04 18:43:17.495432 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.495410    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-dshm\") pod \"02bbacfc-d045-4fe6-8858-11a734763a1c\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") "
Aug 04 18:43:17.495735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.495442    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-tmp-dir\") pod \"02bbacfc-d045-4fe6-8858-11a734763a1c\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") "
Aug 04 18:43:17.495735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.495500    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-model-cache\") pod \"02bbacfc-d045-4fe6-8858-11a734763a1c\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") "
Aug 04 18:43:17.495735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.495560    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhgfz\" (UniqueName: \"kubernetes.io/projected/02bbacfc-d045-4fe6-8858-11a734763a1c-kube-api-access-hhgfz\") pod \"02bbacfc-d045-4fe6-8858-11a734763a1c\" (UID: \"02bbacfc-d045-4fe6-8858-11a734763a1c\") "
Aug 04 18:43:17.495735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.495604    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-home" (OuterVolumeSpecName: "home") pod "02bbacfc-d045-4fe6-8858-11a734763a1c" (UID: "02bbacfc-d045-4fe6-8858-11a734763a1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:43:17.495735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.495709    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "02bbacfc-d045-4fe6-8858-11a734763a1c" (UID: "02bbacfc-d045-4fe6-8858-11a734763a1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:43:17.495988 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.495771    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-model-cache" (OuterVolumeSpecName: "model-cache") pod "02bbacfc-d045-4fe6-8858-11a734763a1c" (UID: "02bbacfc-d045-4fe6-8858-11a734763a1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:43:17.495988 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.495888    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:43:17.495988 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.495907    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:43:17.495988 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.495920    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:43:17.497804 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.497767    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02bbacfc-d045-4fe6-8858-11a734763a1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02bbacfc-d045-4fe6-8858-11a734763a1c" (UID: "02bbacfc-d045-4fe6-8858-11a734763a1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:43:17.497936 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.497803    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bbacfc-d045-4fe6-8858-11a734763a1c-kube-api-access-hhgfz" (OuterVolumeSpecName: "kube-api-access-hhgfz") pod "02bbacfc-d045-4fe6-8858-11a734763a1c" (UID: "02bbacfc-d045-4fe6-8858-11a734763a1c"). InnerVolumeSpecName "kube-api-access-hhgfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:43:17.497936 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.497825    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-dshm" (OuterVolumeSpecName: "dshm") pod "02bbacfc-d045-4fe6-8858-11a734763a1c" (UID: "02bbacfc-d045-4fe6-8858-11a734763a1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:43:17.597310 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.597272    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02bbacfc-d045-4fe6-8858-11a734763a1c-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:43:17.597310 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.597306    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bbacfc-d045-4fe6-8858-11a734763a1c-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:43:17.597310 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.597315    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhgfz\" (UniqueName: \"kubernetes.io/projected/02bbacfc-d045-4fe6-8858-11a734763a1c-kube-api-access-hhgfz\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:43:17.976443 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.976406    2576 generic.go:356] "Generic (PLEG): container finished" podID="02bbacfc-d045-4fe6-8858-11a734763a1c" containerID="6bfb0977f4cc62111784c2cd427b8c87857fbd1525849f27e0354c9605a4e632" exitCode=0
Aug 04 18:43:17.976626 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.976454    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"
Aug 04 18:43:17.976626 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.976497    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m" event={"ID":"02bbacfc-d045-4fe6-8858-11a734763a1c","Type":"ContainerDied","Data":"6bfb0977f4cc62111784c2cd427b8c87857fbd1525849f27e0354c9605a4e632"}
Aug 04 18:43:17.976626 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.976550    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m" event={"ID":"02bbacfc-d045-4fe6-8858-11a734763a1c","Type":"ContainerDied","Data":"153ce39084bfbf6c4a9c91afc745f45bf9d8ad58b9dbdd025f37c44bade4f1df"}
Aug 04 18:43:17.976626 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.976573    2576 scope.go:117] "RemoveContainer" containerID="6bfb0977f4cc62111784c2cd427b8c87857fbd1525849f27e0354c9605a4e632"
Aug 04 18:43:17.985853 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.985830    2576 scope.go:117] "RemoveContainer" containerID="6bfb0977f4cc62111784c2cd427b8c87857fbd1525849f27e0354c9605a4e632"
Aug 04 18:43:17.986172 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:43:17.986121    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfb0977f4cc62111784c2cd427b8c87857fbd1525849f27e0354c9605a4e632\": container with ID starting with 6bfb0977f4cc62111784c2cd427b8c87857fbd1525849f27e0354c9605a4e632 not found: ID does not exist" containerID="6bfb0977f4cc62111784c2cd427b8c87857fbd1525849f27e0354c9605a4e632"
Aug 04 18:43:17.986172 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:17.986153    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfb0977f4cc62111784c2cd427b8c87857fbd1525849f27e0354c9605a4e632"} err="failed to get container status \"6bfb0977f4cc62111784c2cd427b8c87857fbd1525849f27e0354c9605a4e632\": rpc error: code = NotFound desc = could not find container \"6bfb0977f4cc62111784c2cd427b8c87857fbd1525849f27e0354c9605a4e632\": container with ID starting with 6bfb0977f4cc62111784c2cd427b8c87857fbd1525849f27e0354c9605a4e632 not found: ID does not exist"
Aug 04 18:43:18.001086 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:18.001054    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"]
Aug 04 18:43:18.005535 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:18.005503    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-65j4m"]
Aug 04 18:43:18.582446 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:18.582414    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02bbacfc-d045-4fe6-8858-11a734763a1c" path="/var/lib/kubelet/pods/02bbacfc-d045-4fe6-8858-11a734763a1c/volumes"
Aug 04 18:43:22.190397 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:22.190361    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:28.526543 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:28.526507    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:30.259055 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:30.259016    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"]
Aug 04 18:43:30.276996 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:30.276963    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:30.371945 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:30.371903    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:30.543938 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:30.543905    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:30.635541 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:30.635509    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:30.743696 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:30.743654    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:35.254720 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:35.254687    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:35.358764 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:35.358674    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:35.459431 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:35.459396    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:35.774604 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:35.774506    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:38.086482 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:38.086386    2576 generic.go:356] "Generic (PLEG): container finished" podID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerID="2a4d86b0906691e2784674820903cee3b1f7902ca56a02739998b28fb37a3dc3" exitCode=0
Aug 04 18:43:38.086482 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:38.086447    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd" event={"ID":"57b0d619-7e8d-406b-aa0f-923a6e75f33a","Type":"ContainerDied","Data":"2a4d86b0906691e2784674820903cee3b1f7902ca56a02739998b28fb37a3dc3"}
Aug 04 18:43:38.087066 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:38.086910    2576 scope.go:117] "RemoveContainer" containerID="2a4d86b0906691e2784674820903cee3b1f7902ca56a02739998b28fb37a3dc3"
Aug 04 18:43:38.087919 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:38.087893    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:43:38.087975 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:38.087957    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:43:38.088011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:38.087988    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:43:39.092915 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:39.092883    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd" event={"ID":"57b0d619-7e8d-406b-aa0f-923a6e75f33a","Type":"ContainerStarted","Data":"946ccf1ce3622087e8baa99e921687cfc7aa94a6d1d04ab79514a3229aa75260"}
Aug 04 18:43:39.319571 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:39.319534    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:39.569473 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:39.569435    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:43:39.570296 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:39.570270    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd" podUID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Aug 04 18:43:40.238564 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:40.238482    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:40.415649 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:40.415610    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:40.509609 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:40.509525    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:40.569033 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:40.568992    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd" podUID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Aug 04 18:43:41.569892 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:41.569847    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd" podUID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Aug 04 18:43:41.858951 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:41.858799    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:41.955803 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:41.955731    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:42.569962 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:42.569923    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd" podUID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Aug 04 18:43:43.569091 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:43.569051    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd" podUID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Aug 04 18:43:43.882274 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:43.882181    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:43:44.573030 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:44.572997    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:43:44.574007 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:44.573986    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:43:44.969026 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:44.968939    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:45.069308 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:45.069272    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:45.292408 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:45.292368    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb" podUID="20b6a547-3ea8-4cf6-8723-2391b3989fb4" containerName="main" containerID="cri-o://e8bc772ad4803938c30c6cbcd7ad5c6099229a4348c53395a33d0780f10540d3" gracePeriod=45
Aug 04 18:43:45.539770 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:45.539748    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"
Aug 04 18:43:45.552857 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:45.552803    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6zcw\" (UniqueName: \"kubernetes.io/projected/20b6a547-3ea8-4cf6-8723-2391b3989fb4-kube-api-access-d6zcw\") pod \"20b6a547-3ea8-4cf6-8723-2391b3989fb4\" (UID: \"20b6a547-3ea8-4cf6-8723-2391b3989fb4\") "
Aug 04 18:43:45.552857 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:45.552839    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b6a547-3ea8-4cf6-8723-2391b3989fb4-tls-certs\") pod \"20b6a547-3ea8-4cf6-8723-2391b3989fb4\" (UID: \"20b6a547-3ea8-4cf6-8723-2391b3989fb4\") "
Aug 04 18:43:45.554937 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:45.554913    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b6a547-3ea8-4cf6-8723-2391b3989fb4-kube-api-access-d6zcw" (OuterVolumeSpecName: "kube-api-access-d6zcw") pod "20b6a547-3ea8-4cf6-8723-2391b3989fb4" (UID: "20b6a547-3ea8-4cf6-8723-2391b3989fb4"). InnerVolumeSpecName "kube-api-access-d6zcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:43:45.554937 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:45.554923    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20b6a547-3ea8-4cf6-8723-2391b3989fb4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20b6a547-3ea8-4cf6-8723-2391b3989fb4" (UID: "20b6a547-3ea8-4cf6-8723-2391b3989fb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:43:45.654382 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:45.654345    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6zcw\" (UniqueName: \"kubernetes.io/projected/20b6a547-3ea8-4cf6-8723-2391b3989fb4-kube-api-access-d6zcw\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:43:45.654382 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:45.654374    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b6a547-3ea8-4cf6-8723-2391b3989fb4-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:43:46.129423 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:46.129387    2576 generic.go:356] "Generic (PLEG): container finished" podID="20b6a547-3ea8-4cf6-8723-2391b3989fb4" containerID="e8bc772ad4803938c30c6cbcd7ad5c6099229a4348c53395a33d0780f10540d3" exitCode=0
Aug 04 18:43:46.129423 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:46.129423    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"
Aug 04 18:43:46.129886 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:46.129449    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb" event={"ID":"20b6a547-3ea8-4cf6-8723-2391b3989fb4","Type":"ContainerDied","Data":"e8bc772ad4803938c30c6cbcd7ad5c6099229a4348c53395a33d0780f10540d3"}
Aug 04 18:43:46.129886 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:46.129494    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb" event={"ID":"20b6a547-3ea8-4cf6-8723-2391b3989fb4","Type":"ContainerDied","Data":"9804d9e4241fa5d91a65d72ce0c6565e40aea2aebc2fe583029139ce1b3837d6"}
Aug 04 18:43:46.129886 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:46.129515    2576 scope.go:117] "RemoveContainer" containerID="e8bc772ad4803938c30c6cbcd7ad5c6099229a4348c53395a33d0780f10540d3"
Aug 04 18:43:46.138783 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:46.138765    2576 scope.go:117] "RemoveContainer" containerID="e8bc772ad4803938c30c6cbcd7ad5c6099229a4348c53395a33d0780f10540d3"
Aug 04 18:43:46.139028 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:43:46.139012    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8bc772ad4803938c30c6cbcd7ad5c6099229a4348c53395a33d0780f10540d3\": container with ID starting with e8bc772ad4803938c30c6cbcd7ad5c6099229a4348c53395a33d0780f10540d3 not found: ID does not exist" containerID="e8bc772ad4803938c30c6cbcd7ad5c6099229a4348c53395a33d0780f10540d3"
Aug 04 18:43:46.139088 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:46.139036    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8bc772ad4803938c30c6cbcd7ad5c6099229a4348c53395a33d0780f10540d3"} err="failed to get container status \"e8bc772ad4803938c30c6cbcd7ad5c6099229a4348c53395a33d0780f10540d3\": rpc error: code = NotFound desc = could not find container \"e8bc772ad4803938c30c6cbcd7ad5c6099229a4348c53395a33d0780f10540d3\": container with ID starting with e8bc772ad4803938c30c6cbcd7ad5c6099229a4348c53395a33d0780f10540d3 not found: ID does not exist"
Aug 04 18:43:46.154533 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:46.154510    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"]
Aug 04 18:43:46.158222 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:46.158200    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-nzqkb"]
Aug 04 18:43:46.348123 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:46.348072    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:46.452556 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:46.452473    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:46.582567 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:46.582531    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20b6a547-3ea8-4cf6-8723-2391b3989fb4" path="/var/lib/kubelet/pods/20b6a547-3ea8-4cf6-8723-2391b3989fb4/volumes"
Aug 04 18:43:48.334845 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.334804    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:48.520737 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.520700    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"]
Aug 04 18:43:48.521212 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.521192    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20b6a547-3ea8-4cf6-8723-2391b3989fb4" containerName="main"
Aug 04 18:43:48.521212 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.521213    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="20b6a547-3ea8-4cf6-8723-2391b3989fb4" containerName="main"
Aug 04 18:43:48.521410 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.521244    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02bbacfc-d045-4fe6-8858-11a734763a1c" containerName="main"
Aug 04 18:43:48.521410 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.521252    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="02bbacfc-d045-4fe6-8858-11a734763a1c" containerName="main"
Aug 04 18:43:48.521410 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.521344    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="20b6a547-3ea8-4cf6-8723-2391b3989fb4" containerName="main"
Aug 04 18:43:48.521410 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.521363    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="02bbacfc-d045-4fe6-8858-11a734763a1c" containerName="main"
Aug 04 18:43:48.525958 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.525937    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"
Aug 04 18:43:48.529672 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.529649    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-kserve-self-signed-certs\""
Aug 04 18:43:48.529803 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.529741    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Aug 04 18:43:48.529803 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.529786    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Aug 04 18:43:48.529944 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.529796    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-dqwtt\""
Aug 04 18:43:48.529944 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.529656    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-mznbs\""
Aug 04 18:43:48.535589 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.535566    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"]
Aug 04 18:43:48.555960 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.555930    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:48.580173 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.580137    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw\" (UID: \"61559892-abab-4ae0-95ec-932c389ac825\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"
Aug 04 18:43:48.580352 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.580192    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sskb\" (UniqueName: \"kubernetes.io/projected/61559892-abab-4ae0-95ec-932c389ac825-kube-api-access-7sskb\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw\" (UID: \"61559892-abab-4ae0-95ec-932c389ac825\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"
Aug 04 18:43:48.681645 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.681556    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw\" (UID: \"61559892-abab-4ae0-95ec-932c389ac825\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"
Aug 04 18:43:48.681645 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.681607    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sskb\" (UniqueName: \"kubernetes.io/projected/61559892-abab-4ae0-95ec-932c389ac825-kube-api-access-7sskb\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw\" (UID: \"61559892-abab-4ae0-95ec-932c389ac825\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"
Aug 04 18:43:48.684145 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.684088    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw\" (UID: \"61559892-abab-4ae0-95ec-932c389ac825\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"
Aug 04 18:43:48.690438 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.690413    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sskb\" (UniqueName: \"kubernetes.io/projected/61559892-abab-4ae0-95ec-932c389ac825-kube-api-access-7sskb\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw\" (UID: \"61559892-abab-4ae0-95ec-932c389ac825\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"
Aug 04 18:43:48.837607 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.837552    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"
Aug 04 18:43:48.985158 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:48.985131    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"]
Aug 04 18:43:48.986996 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:43:48.986966    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61559892_abab_4ae0_95ec_932c389ac825.slice/crio-7678d7d877a9c23b2a313b3a1a3823ea328df88144bec8a62e7dcb40839206a6 WatchSource:0}: Error finding container 7678d7d877a9c23b2a313b3a1a3823ea328df88144bec8a62e7dcb40839206a6: Status 404 returned error can't find the container with id 7678d7d877a9c23b2a313b3a1a3823ea328df88144bec8a62e7dcb40839206a6
Aug 04 18:43:49.089067 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:49.089028    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:49.147622 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:49.147580    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw" event={"ID":"61559892-abab-4ae0-95ec-932c389ac825","Type":"ContainerStarted","Data":"5425fe4614f97ac66106ac561a8b16e785299278e62333d89b8cedeb037013a4"}
Aug 04 18:43:49.147813 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:49.147655    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw" event={"ID":"61559892-abab-4ae0-95ec-932c389ac825","Type":"ContainerStarted","Data":"7678d7d877a9c23b2a313b3a1a3823ea328df88144bec8a62e7dcb40839206a6"}
Aug 04 18:43:49.148483 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:49.148456    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"
Aug 04 18:43:49.171892 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:49.171835    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw" podStartSLOduration=1.1718197479999999 podStartE2EDuration="1.171819748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:43:49.170527902 +0000 UTC m=+2629.146359599" watchObservedRunningTime="2026-08-04 18:43:49.171819748 +0000 UTC m=+2629.147651445"
Aug 04 18:43:49.175268 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:49.175224    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:50.635725 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:50.635683    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"]
Aug 04 18:43:50.645072 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:50.645015    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:50.705051 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:50.705018    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:51.158637 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:51.158597    2576 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw" secret="" err="secret \"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-mznbs\" not found"
Aug 04 18:43:51.204324 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:43:51.204290    2576 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Aug 04 18:43:51.204507 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:43:51.204387    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs podName:61559892-abab-4ae0-95ec-932c389ac825 nodeName:}" failed. No retries permitted until 2026-08-04 18:43:51.704365465 +0000 UTC m=+2631.680197143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw" (UID: "61559892-abab-4ae0-95ec-932c389ac825") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Aug 04 18:43:51.709314 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:43:51.709282    2576 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Aug 04 18:43:51.709673 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:43:51.709362    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs podName:61559892-abab-4ae0-95ec-932c389ac825 nodeName:}" failed. No retries permitted until 2026-08-04 18:43:52.709345785 +0000 UTC m=+2632.685177461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw" (UID: "61559892-abab-4ae0-95ec-932c389ac825") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Aug 04 18:43:52.721134 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:43:52.721081    2576 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Aug 04 18:43:52.721542 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:43:52.721173    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs podName:61559892-abab-4ae0-95ec-932c389ac825 nodeName:}" failed. No retries permitted until 2026-08-04 18:43:54.721159289 +0000 UTC m=+2634.696990963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw" (UID: "61559892-abab-4ae0-95ec-932c389ac825") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Aug 04 18:43:52.889876 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:52.889841    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:52.984392 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:52.984306    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:54.738289 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:43:54.738246    2576 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Aug 04 18:43:54.738781 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:43:54.738337    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs podName:61559892-abab-4ae0-95ec-932c389ac825 nodeName:}" failed. No retries permitted until 2026-08-04 18:43:58.738320053 +0000 UTC m=+2638.714151729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw" (UID: "61559892-abab-4ae0-95ec-932c389ac825") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Aug 04 18:43:57.947446 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:43:57.947406    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:58.777453 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:43:58.777408    2576 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Aug 04 18:43:58.777674 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:43:58.777496    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs podName:61559892-abab-4ae0-95ec-932c389ac825 nodeName:}" failed. No retries permitted until 2026-08-04 18:44:06.77747741 +0000 UTC m=+2646.753309086 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw" (UID: "61559892-abab-4ae0-95ec-932c389ac825") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Aug 04 18:44:06.858866 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:44:06.858828    2576 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Aug 04 18:44:06.859325 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:44:06.858907    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs podName:61559892-abab-4ae0-95ec-932c389ac825 nodeName:}" failed. No retries permitted until 2026-08-04 18:44:22.858893389 +0000 UTC m=+2662.834725065 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw" (UID: "61559892-abab-4ae0-95ec-932c389ac825") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Aug 04 18:44:07.183242 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:07.183093    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw" podUID="61559892-abab-4ae0-95ec-932c389ac825" containerName="main" containerID="cri-o://5425fe4614f97ac66106ac561a8b16e785299278e62333d89b8cedeb037013a4" gracePeriod=45
Aug 04 18:44:07.440357 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:07.440295    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"
Aug 04 18:44:07.566049 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:07.566013    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sskb\" (UniqueName: \"kubernetes.io/projected/61559892-abab-4ae0-95ec-932c389ac825-kube-api-access-7sskb\") pod \"61559892-abab-4ae0-95ec-932c389ac825\" (UID: \"61559892-abab-4ae0-95ec-932c389ac825\") "
Aug 04 18:44:07.566229 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:07.566088    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs\") pod \"61559892-abab-4ae0-95ec-932c389ac825\" (UID: \"61559892-abab-4ae0-95ec-932c389ac825\") "
Aug 04 18:44:07.568236 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:07.568200    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61559892-abab-4ae0-95ec-932c389ac825" (UID: "61559892-abab-4ae0-95ec-932c389ac825"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:44:07.568236 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:07.568212    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61559892-abab-4ae0-95ec-932c389ac825-kube-api-access-7sskb" (OuterVolumeSpecName: "kube-api-access-7sskb") pod "61559892-abab-4ae0-95ec-932c389ac825" (UID: "61559892-abab-4ae0-95ec-932c389ac825"). InnerVolumeSpecName "kube-api-access-7sskb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:44:07.666710 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:07.666672    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sskb\" (UniqueName: \"kubernetes.io/projected/61559892-abab-4ae0-95ec-932c389ac825-kube-api-access-7sskb\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:44:07.666710 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:07.666703    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61559892-abab-4ae0-95ec-932c389ac825-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:44:08.238164 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:08.238086    2576 generic.go:356] "Generic (PLEG): container finished" podID="61559892-abab-4ae0-95ec-932c389ac825" containerID="5425fe4614f97ac66106ac561a8b16e785299278e62333d89b8cedeb037013a4" exitCode=0
Aug 04 18:44:08.238164 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:08.238149    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"
Aug 04 18:44:08.238738 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:08.238173    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw" event={"ID":"61559892-abab-4ae0-95ec-932c389ac825","Type":"ContainerDied","Data":"5425fe4614f97ac66106ac561a8b16e785299278e62333d89b8cedeb037013a4"}
Aug 04 18:44:08.238738 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:08.238251    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw" event={"ID":"61559892-abab-4ae0-95ec-932c389ac825","Type":"ContainerDied","Data":"7678d7d877a9c23b2a313b3a1a3823ea328df88144bec8a62e7dcb40839206a6"}
Aug 04 18:44:08.238738 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:08.238286    2576 scope.go:117] "RemoveContainer" containerID="5425fe4614f97ac66106ac561a8b16e785299278e62333d89b8cedeb037013a4"
Aug 04 18:44:08.247670 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:08.247653    2576 scope.go:117] "RemoveContainer" containerID="5425fe4614f97ac66106ac561a8b16e785299278e62333d89b8cedeb037013a4"
Aug 04 18:44:08.247925 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:44:08.247905    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5425fe4614f97ac66106ac561a8b16e785299278e62333d89b8cedeb037013a4\": container with ID starting with 5425fe4614f97ac66106ac561a8b16e785299278e62333d89b8cedeb037013a4 not found: ID does not exist" containerID="5425fe4614f97ac66106ac561a8b16e785299278e62333d89b8cedeb037013a4"
Aug 04 18:44:08.247996 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:08.247933    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5425fe4614f97ac66106ac561a8b16e785299278e62333d89b8cedeb037013a4"} err="failed to get container status \"5425fe4614f97ac66106ac561a8b16e785299278e62333d89b8cedeb037013a4\": rpc error: code = NotFound desc = could not find container \"5425fe4614f97ac66106ac561a8b16e785299278e62333d89b8cedeb037013a4\": container with ID starting with 5425fe4614f97ac66106ac561a8b16e785299278e62333d89b8cedeb037013a4 not found: ID does not exist"
Aug 04 18:44:08.260558 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:08.260531    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"]
Aug 04 18:44:08.264321 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:08.264299    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894scfxw"]
Aug 04 18:44:08.582232 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:08.582195    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61559892-abab-4ae0-95ec-932c389ac825" path="/var/lib/kubelet/pods/61559892-abab-4ae0-95ec-932c389ac825/volumes"
Aug 04 18:44:11.122860 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.122817    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:44:11.193656 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.193615    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"]
Aug 04 18:44:11.194086 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.194064    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61559892-abab-4ae0-95ec-932c389ac825" containerName="main"
Aug 04 18:44:11.194086 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.194085    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="61559892-abab-4ae0-95ec-932c389ac825" containerName="main"
Aug 04 18:44:11.194315 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.194173    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="61559892-abab-4ae0-95ec-932c389ac825" containerName="main"
Aug 04 18:44:11.198666 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.198645    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"
Aug 04 18:44:11.205494 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.205468    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-9b65r\""
Aug 04 18:44:11.206758 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.206739    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Aug 04 18:44:11.206897 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.206782    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Aug 04 18:44:11.206897 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.206868    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Aug 04 18:44:11.206989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.206925    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-sncnx\""
Aug 04 18:44:11.215083 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.215057    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"]
Aug 04 18:44:11.232821 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.232777    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:44:11.298309 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.298274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21404599-e6f6-493c-90c8-8798c51226c4-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg\" (UID: \"21404599-e6f6-493c-90c8-8798c51226c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"
Aug 04 18:44:11.298309 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.298310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp8l6\" (UniqueName: \"kubernetes.io/projected/21404599-e6f6-493c-90c8-8798c51226c4-kube-api-access-lp8l6\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg\" (UID: \"21404599-e6f6-493c-90c8-8798c51226c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"
Aug 04 18:44:11.398882 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.398792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21404599-e6f6-493c-90c8-8798c51226c4-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg\" (UID: \"21404599-e6f6-493c-90c8-8798c51226c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"
Aug 04 18:44:11.398882 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.398830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp8l6\" (UniqueName: \"kubernetes.io/projected/21404599-e6f6-493c-90c8-8798c51226c4-kube-api-access-lp8l6\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg\" (UID: \"21404599-e6f6-493c-90c8-8798c51226c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"
Aug 04 18:44:11.401358 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.401336    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21404599-e6f6-493c-90c8-8798c51226c4-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg\" (UID: \"21404599-e6f6-493c-90c8-8798c51226c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"
Aug 04 18:44:11.407563 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.407541    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp8l6\" (UniqueName: \"kubernetes.io/projected/21404599-e6f6-493c-90c8-8798c51226c4-kube-api-access-lp8l6\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg\" (UID: \"21404599-e6f6-493c-90c8-8798c51226c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"
Aug 04 18:44:11.509211 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.509177    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"
Aug 04 18:44:11.645414 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:11.645379    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"]
Aug 04 18:44:11.650969 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:44:11.650904    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21404599_e6f6_493c_90c8_8798c51226c4.slice/crio-628168026d9897389b1718e91cab567c3dd4b6ea472a56c5ad19d8be0345f41d WatchSource:0}: Error finding container 628168026d9897389b1718e91cab567c3dd4b6ea472a56c5ad19d8be0345f41d: Status 404 returned error can't find the container with id 628168026d9897389b1718e91cab567c3dd4b6ea472a56c5ad19d8be0345f41d
Aug 04 18:44:12.261658 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:12.261615    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg" event={"ID":"21404599-e6f6-493c-90c8-8798c51226c4","Type":"ContainerStarted","Data":"cce9bb8864a39b321877860f1307fc16917ae37f415138005b314e8b99807607"}
Aug 04 18:44:12.262092 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:12.261679    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg" event={"ID":"21404599-e6f6-493c-90c8-8798c51226c4","Type":"ContainerStarted","Data":"628168026d9897389b1718e91cab567c3dd4b6ea472a56c5ad19d8be0345f41d"}
Aug 04 18:44:12.262092 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:12.261719    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"
Aug 04 18:44:12.278614 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:12.278562    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg" podStartSLOduration=1.278545716 podStartE2EDuration="1.278545716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:44:12.27785181 +0000 UTC m=+2652.253683507" watchObservedRunningTime="2026-08-04 18:44:12.278545716 +0000 UTC m=+2652.254377437"
Aug 04 18:44:17.453439 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:17.453385    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:44:20.147071 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:20.147035    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:44:20.215387 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:20.215352    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:44:34.881984 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:34.881941    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:44:43.269496 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:43.269462    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"
Aug 04 18:44:43.838983 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:44:43.838951    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:44.662267 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:44.662226    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:49.538313 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:49.538273    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:49.635890 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:49.635853    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:53.451879 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.451843    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:53.713072 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.712980    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"]
Aug 04 18:45:53.716848 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.716822    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"
Aug 04 18:45:53.730740 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.730708    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Aug 04 18:45:53.730909 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.730798    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-mvjvd\""
Aug 04 18:45:53.730909 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.730796    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-rjt7j\""
Aug 04 18:45:53.731049 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.730796    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Aug 04 18:45:53.731049 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.730720    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Aug 04 18:45:53.732809 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.732787    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"]
Aug 04 18:45:53.823726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.823684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s5ft\" (UniqueName: \"kubernetes.io/projected/6daa9c91-2015-4a11-acd0-3506a2bc536b-kube-api-access-4s5ft\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h\" (UID: \"6daa9c91-2015-4a11-acd0-3506a2bc536b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"
Aug 04 18:45:53.823726 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.823729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6daa9c91-2015-4a11-acd0-3506a2bc536b-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h\" (UID: \"6daa9c91-2015-4a11-acd0-3506a2bc536b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"
Aug 04 18:45:53.838451 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.838420    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:53.925280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.925237    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s5ft\" (UniqueName: \"kubernetes.io/projected/6daa9c91-2015-4a11-acd0-3506a2bc536b-kube-api-access-4s5ft\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h\" (UID: \"6daa9c91-2015-4a11-acd0-3506a2bc536b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"
Aug 04 18:45:53.925280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.925281    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6daa9c91-2015-4a11-acd0-3506a2bc536b-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h\" (UID: \"6daa9c91-2015-4a11-acd0-3506a2bc536b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"
Aug 04 18:45:53.927917 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.927887    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6daa9c91-2015-4a11-acd0-3506a2bc536b-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h\" (UID: \"6daa9c91-2015-4a11-acd0-3506a2bc536b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"
Aug 04 18:45:53.939735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:53.939707    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s5ft\" (UniqueName: \"kubernetes.io/projected/6daa9c91-2015-4a11-acd0-3506a2bc536b-kube-api-access-4s5ft\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h\" (UID: \"6daa9c91-2015-4a11-acd0-3506a2bc536b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"
Aug 04 18:45:54.028070 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:54.027969    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"
Aug 04 18:45:54.168571 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:54.168539    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"]
Aug 04 18:45:54.172361 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:45:54.172328    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6daa9c91_2015_4a11_acd0_3506a2bc536b.slice/crio-1956410a1972813ed9153646d110b6e44dc8b5747a39700f5082ad9599b3b4dc WatchSource:0}: Error finding container 1956410a1972813ed9153646d110b6e44dc8b5747a39700f5082ad9599b3b4dc: Status 404 returned error can't find the container with id 1956410a1972813ed9153646d110b6e44dc8b5747a39700f5082ad9599b3b4dc
Aug 04 18:45:54.491995 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:54.491953    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"]
Aug 04 18:45:54.492500 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:54.492256    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd" podUID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerName="istio-proxy" containerID="cri-o://946ccf1ce3622087e8baa99e921687cfc7aa94a6d1d04ab79514a3229aa75260" gracePeriod=30
Aug 04 18:45:54.586773 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:54.586721    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:54.746709 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:54.746618    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h" event={"ID":"6daa9c91-2015-4a11-acd0-3506a2bc536b","Type":"ContainerStarted","Data":"ed7176f0e8bb23061d3d3f80bb5fd37220b0c9bc2d3ead6d38bb23e70d7de410"}
Aug 04 18:45:54.746709 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:54.746689    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"
Aug 04 18:45:54.746902 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:54.746714    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h" event={"ID":"6daa9c91-2015-4a11-acd0-3506a2bc536b","Type":"ContainerStarted","Data":"1956410a1972813ed9153646d110b6e44dc8b5747a39700f5082ad9599b3b4dc"}
Aug 04 18:45:54.779771 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:54.779705    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h" podStartSLOduration=1.779689691 podStartE2EDuration="1.779689691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:45:54.778803333 +0000 UTC m=+2754.754635032" watchObservedRunningTime="2026-08-04 18:45:54.779689691 +0000 UTC m=+2754.755521387"
Aug 04 18:45:59.574669 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.574631    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd" podUID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Aug 04 18:45:59.756811 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.756785    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:45:59.771641 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.771554    2576 generic.go:356] "Generic (PLEG): container finished" podID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerID="946ccf1ce3622087e8baa99e921687cfc7aa94a6d1d04ab79514a3229aa75260" exitCode=0
Aug 04 18:45:59.771641 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.771600    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"
Aug 04 18:45:59.771641 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.771607    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd" event={"ID":"57b0d619-7e8d-406b-aa0f-923a6e75f33a","Type":"ContainerDied","Data":"946ccf1ce3622087e8baa99e921687cfc7aa94a6d1d04ab79514a3229aa75260"}
Aug 04 18:45:59.771877 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.771654    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd" event={"ID":"57b0d619-7e8d-406b-aa0f-923a6e75f33a","Type":"ContainerDied","Data":"0d3155681ec1f832d484d029fb72e3695b5fcfdd8501a1861b312638445b1168"}
Aug 04 18:45:59.771877 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.771680    2576 scope.go:117] "RemoveContainer" containerID="946ccf1ce3622087e8baa99e921687cfc7aa94a6d1d04ab79514a3229aa75260"
Aug 04 18:45:59.775902 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.775879    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istiod-ca-cert\") pod \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") "
Aug 04 18:45:59.776019 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.775911    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-workload-certs\") pod \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") "
Aug 04 18:45:59.776019 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.775947    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-envoy\") pod \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") "
Aug 04 18:45:59.776019 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.775979    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-workload-socket\") pod \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") "
Aug 04 18:45:59.776285 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.776046    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-token\") pod \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") "
Aug 04 18:45:59.776285 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.776096    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-data\") pod \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") "
Aug 04 18:45:59.776285 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.776138    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-credential-socket\") pod \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") "
Aug 04 18:45:59.776285 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.776181    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74jk7\" (UniqueName: \"kubernetes.io/projected/57b0d619-7e8d-406b-aa0f-923a6e75f33a-kube-api-access-74jk7\") pod \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") "
Aug 04 18:45:59.776285 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.776198    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "57b0d619-7e8d-406b-aa0f-923a6e75f33a" (UID: "57b0d619-7e8d-406b-aa0f-923a6e75f33a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:45:59.776285 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.776220    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-podinfo\") pod \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\" (UID: \"57b0d619-7e8d-406b-aa0f-923a6e75f33a\") "
Aug 04 18:45:59.776611 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.776437    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "57b0d619-7e8d-406b-aa0f-923a6e75f33a" (UID: "57b0d619-7e8d-406b-aa0f-923a6e75f33a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:45:59.776611 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.776474    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "57b0d619-7e8d-406b-aa0f-923a6e75f33a" (UID: "57b0d619-7e8d-406b-aa0f-923a6e75f33a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:45:59.776611 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.776524    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:45:59.776611 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.776544    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:45:59.776611 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.776605    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "57b0d619-7e8d-406b-aa0f-923a6e75f33a" (UID: "57b0d619-7e8d-406b-aa0f-923a6e75f33a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:45:59.776851 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.776825    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-data" (OuterVolumeSpecName: "istio-data") pod "57b0d619-7e8d-406b-aa0f-923a6e75f33a" (UID: "57b0d619-7e8d-406b-aa0f-923a6e75f33a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:45:59.779332 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.779291    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "57b0d619-7e8d-406b-aa0f-923a6e75f33a" (UID: "57b0d619-7e8d-406b-aa0f-923a6e75f33a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 04 18:45:59.779488 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.779459    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b0d619-7e8d-406b-aa0f-923a6e75f33a-kube-api-access-74jk7" (OuterVolumeSpecName: "kube-api-access-74jk7") pod "57b0d619-7e8d-406b-aa0f-923a6e75f33a" (UID: "57b0d619-7e8d-406b-aa0f-923a6e75f33a"). InnerVolumeSpecName "kube-api-access-74jk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:45:59.779559 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.779489    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-token" (OuterVolumeSpecName: "istio-token") pod "57b0d619-7e8d-406b-aa0f-923a6e75f33a" (UID: "57b0d619-7e8d-406b-aa0f-923a6e75f33a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:45:59.779803 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.779781    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "57b0d619-7e8d-406b-aa0f-923a6e75f33a" (UID: "57b0d619-7e8d-406b-aa0f-923a6e75f33a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:45:59.783167 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.783140    2576 scope.go:117] "RemoveContainer" containerID="2a4d86b0906691e2784674820903cee3b1f7902ca56a02739998b28fb37a3dc3"
Aug 04 18:45:59.803024 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.803000    2576 scope.go:117] "RemoveContainer" containerID="946ccf1ce3622087e8baa99e921687cfc7aa94a6d1d04ab79514a3229aa75260"
Aug 04 18:45:59.803424 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:45:59.803399    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946ccf1ce3622087e8baa99e921687cfc7aa94a6d1d04ab79514a3229aa75260\": container with ID starting with 946ccf1ce3622087e8baa99e921687cfc7aa94a6d1d04ab79514a3229aa75260 not found: ID does not exist" containerID="946ccf1ce3622087e8baa99e921687cfc7aa94a6d1d04ab79514a3229aa75260"
Aug 04 18:45:59.803500 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.803433    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946ccf1ce3622087e8baa99e921687cfc7aa94a6d1d04ab79514a3229aa75260"} err="failed to get container status \"946ccf1ce3622087e8baa99e921687cfc7aa94a6d1d04ab79514a3229aa75260\": rpc error: code = NotFound desc = could not find container \"946ccf1ce3622087e8baa99e921687cfc7aa94a6d1d04ab79514a3229aa75260\": container with ID starting with 946ccf1ce3622087e8baa99e921687cfc7aa94a6d1d04ab79514a3229aa75260 not found: ID does not exist"
Aug 04 18:45:59.803500 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.803454    2576 scope.go:117] "RemoveContainer" containerID="2a4d86b0906691e2784674820903cee3b1f7902ca56a02739998b28fb37a3dc3"
Aug 04 18:45:59.803764 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:45:59.803745    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4d86b0906691e2784674820903cee3b1f7902ca56a02739998b28fb37a3dc3\": container with ID starting with 2a4d86b0906691e2784674820903cee3b1f7902ca56a02739998b28fb37a3dc3 not found: ID does not exist" containerID="2a4d86b0906691e2784674820903cee3b1f7902ca56a02739998b28fb37a3dc3"
Aug 04 18:45:59.803820 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.803774    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4d86b0906691e2784674820903cee3b1f7902ca56a02739998b28fb37a3dc3"} err="failed to get container status \"2a4d86b0906691e2784674820903cee3b1f7902ca56a02739998b28fb37a3dc3\": rpc error: code = NotFound desc = could not find container \"2a4d86b0906691e2784674820903cee3b1f7902ca56a02739998b28fb37a3dc3\": container with ID starting with 2a4d86b0906691e2784674820903cee3b1f7902ca56a02739998b28fb37a3dc3 not found: ID does not exist"
Aug 04 18:45:59.877737 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.877689    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:45:59.877737 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.877732    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:45:59.877737 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.877743    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74jk7\" (UniqueName: \"kubernetes.io/projected/57b0d619-7e8d-406b-aa0f-923a6e75f33a-kube-api-access-74jk7\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:45:59.877737 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.877754    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:45:59.878003 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.877763    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:45:59.878003 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.877771    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:45:59.878003 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:45:59.877779    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b0d619-7e8d-406b-aa0f-923a6e75f33a-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:46:00.093373 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:00.093334    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"]
Aug 04 18:46:00.097399 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:00.097355    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-77cc9b855f-s9pmd"]
Aug 04 18:46:00.583591 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:00.583557    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" path="/var/lib/kubelet/pods/57b0d619-7e8d-406b-aa0f-923a6e75f33a/volumes"
Aug 04 18:46:12.467396 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:12.467353    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:15.506458 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:15.506425    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:15.569463 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:15.569430    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"]
Aug 04 18:46:15.606380 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:15.606342    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:15.706701 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:15.706664    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:15.808955 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:15.808919    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:17.361185 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:17.361150    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:23.268546 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:23.268451    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg" podUID="21404599-e6f6-493c-90c8-8798c51226c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:46:24.222890 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:24.222855    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:25.753746 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:25.753710    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"
Aug 04 18:46:30.588998 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:30.588961    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg" podUID="21404599-e6f6-493c-90c8-8798c51226c4" containerName="main" containerID="cri-o://cce9bb8864a39b321877860f1307fc16917ae37f415138005b314e8b99807607" gracePeriod=45
Aug 04 18:46:30.838386 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:30.838340    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:30.841640 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:30.841579    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"
Aug 04 18:46:30.926366 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:30.926333    2576 generic.go:356] "Generic (PLEG): container finished" podID="21404599-e6f6-493c-90c8-8798c51226c4" containerID="cce9bb8864a39b321877860f1307fc16917ae37f415138005b314e8b99807607" exitCode=0
Aug 04 18:46:30.926537 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:30.926370    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"
Aug 04 18:46:30.926537 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:30.926372    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg" event={"ID":"21404599-e6f6-493c-90c8-8798c51226c4","Type":"ContainerDied","Data":"cce9bb8864a39b321877860f1307fc16917ae37f415138005b314e8b99807607"}
Aug 04 18:46:30.926537 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:30.926497    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg" event={"ID":"21404599-e6f6-493c-90c8-8798c51226c4","Type":"ContainerDied","Data":"628168026d9897389b1718e91cab567c3dd4b6ea472a56c5ad19d8be0345f41d"}
Aug 04 18:46:30.926537 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:30.926524    2576 scope.go:117] "RemoveContainer" containerID="cce9bb8864a39b321877860f1307fc16917ae37f415138005b314e8b99807607"
Aug 04 18:46:30.934658 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:30.934639    2576 scope.go:117] "RemoveContainer" containerID="cce9bb8864a39b321877860f1307fc16917ae37f415138005b314e8b99807607"
Aug 04 18:46:30.934895 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:46:30.934877    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce9bb8864a39b321877860f1307fc16917ae37f415138005b314e8b99807607\": container with ID starting with cce9bb8864a39b321877860f1307fc16917ae37f415138005b314e8b99807607 not found: ID does not exist" containerID="cce9bb8864a39b321877860f1307fc16917ae37f415138005b314e8b99807607"
Aug 04 18:46:30.934942 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:30.934905    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce9bb8864a39b321877860f1307fc16917ae37f415138005b314e8b99807607"} err="failed to get container status \"cce9bb8864a39b321877860f1307fc16917ae37f415138005b314e8b99807607\": rpc error: code = NotFound desc = could not find container \"cce9bb8864a39b321877860f1307fc16917ae37f415138005b314e8b99807607\": container with ID starting with cce9bb8864a39b321877860f1307fc16917ae37f415138005b314e8b99807607 not found: ID does not exist"
Aug 04 18:46:30.971433 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:30.971385    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21404599-e6f6-493c-90c8-8798c51226c4-tls-certs\") pod \"21404599-e6f6-493c-90c8-8798c51226c4\" (UID: \"21404599-e6f6-493c-90c8-8798c51226c4\") "
Aug 04 18:46:30.971593 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:30.971581    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp8l6\" (UniqueName: \"kubernetes.io/projected/21404599-e6f6-493c-90c8-8798c51226c4-kube-api-access-lp8l6\") pod \"21404599-e6f6-493c-90c8-8798c51226c4\" (UID: \"21404599-e6f6-493c-90c8-8798c51226c4\") "
Aug 04 18:46:30.973651 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:30.973625    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21404599-e6f6-493c-90c8-8798c51226c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21404599-e6f6-493c-90c8-8798c51226c4" (UID: "21404599-e6f6-493c-90c8-8798c51226c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:46:30.973762 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:30.973627    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21404599-e6f6-493c-90c8-8798c51226c4-kube-api-access-lp8l6" (OuterVolumeSpecName: "kube-api-access-lp8l6") pod "21404599-e6f6-493c-90c8-8798c51226c4" (UID: "21404599-e6f6-493c-90c8-8798c51226c4"). InnerVolumeSpecName "kube-api-access-lp8l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:46:31.073083 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.073043    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp8l6\" (UniqueName: \"kubernetes.io/projected/21404599-e6f6-493c-90c8-8798c51226c4-kube-api-access-lp8l6\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:46:31.073083 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.073075    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21404599-e6f6-493c-90c8-8798c51226c4-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:46:31.123840 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.123751    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"]
Aug 04 18:46:31.124240 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.124221    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerName="istio-proxy"
Aug 04 18:46:31.124344 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.124241    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerName="istio-proxy"
Aug 04 18:46:31.124344 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.124255    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21404599-e6f6-493c-90c8-8798c51226c4" containerName="main"
Aug 04 18:46:31.124344 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.124264    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="21404599-e6f6-493c-90c8-8798c51226c4" containerName="main"
Aug 04 18:46:31.124545 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.124382    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerName="istio-proxy"
Aug 04 18:46:31.124545 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.124396    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="21404599-e6f6-493c-90c8-8798c51226c4" containerName="main"
Aug 04 18:46:31.128682 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.128661    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"
Aug 04 18:46:31.131191 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.131170    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-gp4x5\""
Aug 04 18:46:31.138680 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.138636    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"]
Aug 04 18:46:31.146544 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.146516    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:31.250840 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.250809    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"]
Aug 04 18:46:31.255061 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.255034    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-267rg"]
Aug 04 18:46:31.274841 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.274801    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjwp\" (UniqueName: \"kubernetes.io/projected/3c1714e8-afdf-4b1b-9595-d227bb488d84-kube-api-access-8vjwp\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l\" (UID: \"3c1714e8-afdf-4b1b-9595-d227bb488d84\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"
Aug 04 18:46:31.275009 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.274866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c1714e8-afdf-4b1b-9595-d227bb488d84-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l\" (UID: \"3c1714e8-afdf-4b1b-9595-d227bb488d84\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"
Aug 04 18:46:31.376419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.376326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjwp\" (UniqueName: \"kubernetes.io/projected/3c1714e8-afdf-4b1b-9595-d227bb488d84-kube-api-access-8vjwp\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l\" (UID: \"3c1714e8-afdf-4b1b-9595-d227bb488d84\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"
Aug 04 18:46:31.376419 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.376398    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c1714e8-afdf-4b1b-9595-d227bb488d84-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l\" (UID: \"3c1714e8-afdf-4b1b-9595-d227bb488d84\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"
Aug 04 18:46:31.378863 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.378843    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c1714e8-afdf-4b1b-9595-d227bb488d84-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l\" (UID: \"3c1714e8-afdf-4b1b-9595-d227bb488d84\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"
Aug 04 18:46:31.385732 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.385711    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjwp\" (UniqueName: \"kubernetes.io/projected/3c1714e8-afdf-4b1b-9595-d227bb488d84-kube-api-access-8vjwp\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l\" (UID: \"3c1714e8-afdf-4b1b-9595-d227bb488d84\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"
Aug 04 18:46:31.439630 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.439589    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"
Aug 04 18:46:31.575695 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.575664    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"]
Aug 04 18:46:31.577657 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:46:31.577635    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1714e8_afdf_4b1b_9595_d227bb488d84.slice/crio-0a8f6378753438ef3140fdc9f41b9823cbc04642b14fbbdcf100f30e5216660c WatchSource:0}: Error finding container 0a8f6378753438ef3140fdc9f41b9823cbc04642b14fbbdcf100f30e5216660c: Status 404 returned error can't find the container with id 0a8f6378753438ef3140fdc9f41b9823cbc04642b14fbbdcf100f30e5216660c
Aug 04 18:46:31.933966 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.933873    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l" event={"ID":"3c1714e8-afdf-4b1b-9595-d227bb488d84","Type":"ContainerStarted","Data":"d45f0376ea33f41c0779e86bae4b24de6624e5a572fb15a94fff7f2d55374a06"}
Aug 04 18:46:31.933966 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.933928    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l" event={"ID":"3c1714e8-afdf-4b1b-9595-d227bb488d84","Type":"ContainerStarted","Data":"0a8f6378753438ef3140fdc9f41b9823cbc04642b14fbbdcf100f30e5216660c"}
Aug 04 18:46:31.934423 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.933970    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"
Aug 04 18:46:31.955122 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:31.955068    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l" podStartSLOduration=0.95505355 podStartE2EDuration="955.05355ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:46:31.95331178 +0000 UTC m=+2791.929143478" watchObservedRunningTime="2026-08-04 18:46:31.95505355 +0000 UTC m=+2791.930885248"
Aug 04 18:46:32.583004 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:32.582968    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21404599-e6f6-493c-90c8-8798c51226c4" path="/var/lib/kubelet/pods/21404599-e6f6-493c-90c8-8798c51226c4/volumes"
Aug 04 18:46:36.014333 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:36.014284    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:42.096684 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:42.096647    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:45.972989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:45.972958    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:46.073514 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:46.073478    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:58.828989 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:46:58.828947    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:47:02.940928 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:47:02.940893    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"
Aug 04 18:47:58.913105 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:47:58.913018    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:47:58.962378 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:47:58.961960    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"]
Aug 04 18:48:00.262897 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:00.262859    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:48:05.753228 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:05.753183    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h" podUID="6daa9c91-2015-4a11-acd0-3506a2bc536b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:48:13.985752 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:13.985709    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h" podUID="6daa9c91-2015-4a11-acd0-3506a2bc536b" containerName="main" containerID="cri-o://ed7176f0e8bb23061d3d3f80bb5fd37220b0c9bc2d3ead6d38bb23e70d7de410" gracePeriod=45
Aug 04 18:48:14.239814 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.239752    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"
Aug 04 18:48:14.302719 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.302683    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6daa9c91-2015-4a11-acd0-3506a2bc536b-tls-certs\") pod \"6daa9c91-2015-4a11-acd0-3506a2bc536b\" (UID: \"6daa9c91-2015-4a11-acd0-3506a2bc536b\") "
Aug 04 18:48:14.302900 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.302795    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s5ft\" (UniqueName: \"kubernetes.io/projected/6daa9c91-2015-4a11-acd0-3506a2bc536b-kube-api-access-4s5ft\") pod \"6daa9c91-2015-4a11-acd0-3506a2bc536b\" (UID: \"6daa9c91-2015-4a11-acd0-3506a2bc536b\") "
Aug 04 18:48:14.304876 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.304838    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6daa9c91-2015-4a11-acd0-3506a2bc536b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6daa9c91-2015-4a11-acd0-3506a2bc536b" (UID: "6daa9c91-2015-4a11-acd0-3506a2bc536b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:48:14.304991 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.304895    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6daa9c91-2015-4a11-acd0-3506a2bc536b-kube-api-access-4s5ft" (OuterVolumeSpecName: "kube-api-access-4s5ft") pod "6daa9c91-2015-4a11-acd0-3506a2bc536b" (UID: "6daa9c91-2015-4a11-acd0-3506a2bc536b"). InnerVolumeSpecName "kube-api-access-4s5ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:48:14.404412 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.404373    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6daa9c91-2015-4a11-acd0-3506a2bc536b-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:48:14.404412 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.404404    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s5ft\" (UniqueName: \"kubernetes.io/projected/6daa9c91-2015-4a11-acd0-3506a2bc536b-kube-api-access-4s5ft\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:48:14.419576 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.419544    2576 generic.go:356] "Generic (PLEG): container finished" podID="6daa9c91-2015-4a11-acd0-3506a2bc536b" containerID="ed7176f0e8bb23061d3d3f80bb5fd37220b0c9bc2d3ead6d38bb23e70d7de410" exitCode=0
Aug 04 18:48:14.419735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.419587    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"
Aug 04 18:48:14.419735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.419596    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h" event={"ID":"6daa9c91-2015-4a11-acd0-3506a2bc536b","Type":"ContainerDied","Data":"ed7176f0e8bb23061d3d3f80bb5fd37220b0c9bc2d3ead6d38bb23e70d7de410"}
Aug 04 18:48:14.419735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.419647    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h" event={"ID":"6daa9c91-2015-4a11-acd0-3506a2bc536b","Type":"ContainerDied","Data":"1956410a1972813ed9153646d110b6e44dc8b5747a39700f5082ad9599b3b4dc"}
Aug 04 18:48:14.419735 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.419672    2576 scope.go:117] "RemoveContainer" containerID="ed7176f0e8bb23061d3d3f80bb5fd37220b0c9bc2d3ead6d38bb23e70d7de410"
Aug 04 18:48:14.429071 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.429050    2576 scope.go:117] "RemoveContainer" containerID="ed7176f0e8bb23061d3d3f80bb5fd37220b0c9bc2d3ead6d38bb23e70d7de410"
Aug 04 18:48:14.429386 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:48:14.429362    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7176f0e8bb23061d3d3f80bb5fd37220b0c9bc2d3ead6d38bb23e70d7de410\": container with ID starting with ed7176f0e8bb23061d3d3f80bb5fd37220b0c9bc2d3ead6d38bb23e70d7de410 not found: ID does not exist" containerID="ed7176f0e8bb23061d3d3f80bb5fd37220b0c9bc2d3ead6d38bb23e70d7de410"
Aug 04 18:48:14.429462 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.429393    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7176f0e8bb23061d3d3f80bb5fd37220b0c9bc2d3ead6d38bb23e70d7de410"} err="failed to get container status \"ed7176f0e8bb23061d3d3f80bb5fd37220b0c9bc2d3ead6d38bb23e70d7de410\": rpc error: code = NotFound desc = could not find container \"ed7176f0e8bb23061d3d3f80bb5fd37220b0c9bc2d3ead6d38bb23e70d7de410\": container with ID starting with ed7176f0e8bb23061d3d3f80bb5fd37220b0c9bc2d3ead6d38bb23e70d7de410 not found: ID does not exist"
Aug 04 18:48:14.449894 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.449860    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"]
Aug 04 18:48:14.460804 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.460758    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepxm2h"]
Aug 04 18:48:14.582529 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:14.582487    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6daa9c91-2015-4a11-acd0-3506a2bc536b" path="/var/lib/kubelet/pods/6daa9c91-2015-4a11-acd0-3506a2bc536b/volumes"
Aug 04 18:48:18.348466 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:18.348434    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:48:18.624944 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:18.624851    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:48:18.717033 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:18.717002    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:48:24.868750 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:24.868705    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:48:30.565315 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:30.565270    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:48:30.625537 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:30.625500    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:48:46.417858 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.417818    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:48:46.601130 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.601073    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"]
Aug 04 18:48:46.602088 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.602060    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6daa9c91-2015-4a11-acd0-3506a2bc536b" containerName="main"
Aug 04 18:48:46.602218 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.602090    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6daa9c91-2015-4a11-acd0-3506a2bc536b" containerName="main"
Aug 04 18:48:46.602218 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.602192    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerName="istio-proxy"
Aug 04 18:48:46.602218 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.602205    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerName="istio-proxy"
Aug 04 18:48:46.602668 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.602637    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6daa9c91-2015-4a11-acd0-3506a2bc536b" containerName="main"
Aug 04 18:48:46.602756 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.602669    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="57b0d619-7e8d-406b-aa0f-923a6e75f33a" containerName="istio-proxy"
Aug 04 18:48:46.612337 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.612306    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"
Aug 04 18:48:46.614262 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.614237    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"]
Aug 04 18:48:46.616366 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.616337    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-m6frn\""
Aug 04 18:48:46.616480 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.616411    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-vxv4p\""
Aug 04 18:48:46.616480 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.616473    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Aug 04 18:48:46.616661 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.616642    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Aug 04 18:48:46.616803 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.616662    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Aug 04 18:48:46.638565 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.638535    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:48:46.708241 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.708165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc6pv\" (UniqueName: \"kubernetes.io/projected/e0c034b6-0600-430b-aafa-f29fe15abb40-kube-api-access-zc6pv\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7\" (UID: \"e0c034b6-0600-430b-aafa-f29fe15abb40\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"
Aug 04 18:48:46.708241 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.708220    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c034b6-0600-430b-aafa-f29fe15abb40-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7\" (UID: \"e0c034b6-0600-430b-aafa-f29fe15abb40\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"
Aug 04 18:48:46.809573 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.809541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc6pv\" (UniqueName: \"kubernetes.io/projected/e0c034b6-0600-430b-aafa-f29fe15abb40-kube-api-access-zc6pv\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7\" (UID: \"e0c034b6-0600-430b-aafa-f29fe15abb40\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"
Aug 04 18:48:46.809770 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.809586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c034b6-0600-430b-aafa-f29fe15abb40-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7\" (UID: \"e0c034b6-0600-430b-aafa-f29fe15abb40\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"
Aug 04 18:48:46.812014 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.811985    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c034b6-0600-430b-aafa-f29fe15abb40-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7\" (UID: \"e0c034b6-0600-430b-aafa-f29fe15abb40\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"
Aug 04 18:48:46.819509 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.819477    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc6pv\" (UniqueName: \"kubernetes.io/projected/e0c034b6-0600-430b-aafa-f29fe15abb40-kube-api-access-zc6pv\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7\" (UID: \"e0c034b6-0600-430b-aafa-f29fe15abb40\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"
Aug 04 18:48:46.924206 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:46.924168    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"
Aug 04 18:48:47.073908 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:47.073874    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"]
Aug 04 18:48:47.075178 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:48:47.075146    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c034b6_0600_430b_aafa_f29fe15abb40.slice/crio-c8de8e77f7131d6de5cc1961863ede2d106d1b36f645c64dfb4d60684a89acb3 WatchSource:0}: Error finding container c8de8e77f7131d6de5cc1961863ede2d106d1b36f645c64dfb4d60684a89acb3: Status 404 returned error can't find the container with id c8de8e77f7131d6de5cc1961863ede2d106d1b36f645c64dfb4d60684a89acb3
Aug 04 18:48:47.583191 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:47.583155    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7" event={"ID":"e0c034b6-0600-430b-aafa-f29fe15abb40","Type":"ContainerStarted","Data":"04f5a79a55d349480be083a35a8dc72981ed4a0cf76b10d3436ce893f3f35cf6"}
Aug 04 18:48:47.583599 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:47.583215    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7" event={"ID":"e0c034b6-0600-430b-aafa-f29fe15abb40","Type":"ContainerStarted","Data":"c8de8e77f7131d6de5cc1961863ede2d106d1b36f645c64dfb4d60684a89acb3"}
Aug 04 18:48:47.583599 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:47.583254    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"
Aug 04 18:48:47.603283 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:48:47.603235    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7" podStartSLOduration=1.6032183199999999 podStartE2EDuration="1.60321832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:48:47.602052836 +0000 UTC m=+2927.577884561" watchObservedRunningTime="2026-08-04 18:48:47.60321832 +0000 UTC m=+2927.579050018"
Aug 04 18:49:05.079697 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:05.079659    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:06.007969 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:06.007904    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:06.021546 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:06.021508    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"]
Aug 04 18:49:06.148858 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:06.148819    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:12.940379 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:12.940337    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l" podUID="3c1714e8-afdf-4b1b-9595-d227bb488d84" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:49:17.377410 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:17.377365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:18.589258 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:18.589228    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"
Aug 04 18:49:21.048180 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.048136    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l" podUID="3c1714e8-afdf-4b1b-9595-d227bb488d84" containerName="main" containerID="cri-o://d45f0376ea33f41c0779e86bae4b24de6624e5a572fb15a94fff7f2d55374a06" gracePeriod=45
Aug 04 18:49:21.298398 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.298338    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"
Aug 04 18:49:21.431767 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.431732    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c1714e8-afdf-4b1b-9595-d227bb488d84-tls-certs\") pod \"3c1714e8-afdf-4b1b-9595-d227bb488d84\" (UID: \"3c1714e8-afdf-4b1b-9595-d227bb488d84\") "
Aug 04 18:49:21.431940 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.431809    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vjwp\" (UniqueName: \"kubernetes.io/projected/3c1714e8-afdf-4b1b-9595-d227bb488d84-kube-api-access-8vjwp\") pod \"3c1714e8-afdf-4b1b-9595-d227bb488d84\" (UID: \"3c1714e8-afdf-4b1b-9595-d227bb488d84\") "
Aug 04 18:49:21.433921 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.433887    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1714e8-afdf-4b1b-9595-d227bb488d84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c1714e8-afdf-4b1b-9595-d227bb488d84" (UID: "3c1714e8-afdf-4b1b-9595-d227bb488d84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:49:21.434044 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.433926    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1714e8-afdf-4b1b-9595-d227bb488d84-kube-api-access-8vjwp" (OuterVolumeSpecName: "kube-api-access-8vjwp") pod "3c1714e8-afdf-4b1b-9595-d227bb488d84" (UID: "3c1714e8-afdf-4b1b-9595-d227bb488d84"). InnerVolumeSpecName "kube-api-access-8vjwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:49:21.533359 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.533309    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c1714e8-afdf-4b1b-9595-d227bb488d84-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:49:21.533359 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.533353    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vjwp\" (UniqueName: \"kubernetes.io/projected/3c1714e8-afdf-4b1b-9595-d227bb488d84-kube-api-access-8vjwp\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:49:21.753939 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.753853    2576 generic.go:356] "Generic (PLEG): container finished" podID="3c1714e8-afdf-4b1b-9595-d227bb488d84" containerID="d45f0376ea33f41c0779e86bae4b24de6624e5a572fb15a94fff7f2d55374a06" exitCode=0
Aug 04 18:49:21.753939 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.753897    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"
Aug 04 18:49:21.754136 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.753941    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l" event={"ID":"3c1714e8-afdf-4b1b-9595-d227bb488d84","Type":"ContainerDied","Data":"d45f0376ea33f41c0779e86bae4b24de6624e5a572fb15a94fff7f2d55374a06"}
Aug 04 18:49:21.754136 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.753993    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l" event={"ID":"3c1714e8-afdf-4b1b-9595-d227bb488d84","Type":"ContainerDied","Data":"0a8f6378753438ef3140fdc9f41b9823cbc04642b14fbbdcf100f30e5216660c"}
Aug 04 18:49:21.754136 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.754015    2576 scope.go:117] "RemoveContainer" containerID="d45f0376ea33f41c0779e86bae4b24de6624e5a572fb15a94fff7f2d55374a06"
Aug 04 18:49:21.762837 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.762818    2576 scope.go:117] "RemoveContainer" containerID="d45f0376ea33f41c0779e86bae4b24de6624e5a572fb15a94fff7f2d55374a06"
Aug 04 18:49:21.763083 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:49:21.763060    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45f0376ea33f41c0779e86bae4b24de6624e5a572fb15a94fff7f2d55374a06\": container with ID starting with d45f0376ea33f41c0779e86bae4b24de6624e5a572fb15a94fff7f2d55374a06 not found: ID does not exist" containerID="d45f0376ea33f41c0779e86bae4b24de6624e5a572fb15a94fff7f2d55374a06"
Aug 04 18:49:21.763192 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.763090    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45f0376ea33f41c0779e86bae4b24de6624e5a572fb15a94fff7f2d55374a06"} err="failed to get container status \"d45f0376ea33f41c0779e86bae4b24de6624e5a572fb15a94fff7f2d55374a06\": rpc error: code = NotFound desc = could not find container \"d45f0376ea33f41c0779e86bae4b24de6624e5a572fb15a94fff7f2d55374a06\": container with ID starting with d45f0376ea33f41c0779e86bae4b24de6624e5a572fb15a94fff7f2d55374a06 not found: ID does not exist"
Aug 04 18:49:21.776908 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.776875    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"]
Aug 04 18:49:21.781190 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:21.781168    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gdf4l"]
Aug 04 18:49:22.583417 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:22.583342    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c1714e8-afdf-4b1b-9595-d227bb488d84" path="/var/lib/kubelet/pods/3c1714e8-afdf-4b1b-9595-d227bb488d84/volumes"
Aug 04 18:49:25.486712 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:25.486680    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:25.587576 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:25.587537    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:25.686218 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:25.686169    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:31.627808 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:31.627745    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:37.473200 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:37.473168    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:37.660083 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:37.660048    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:44.543334 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:49:44.543302    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:51.253394 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:50:51.253360    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"]
Aug 04 18:50:51.262272 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:50:51.262237    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:51.319634 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:50:51.319593    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:58.588717 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:50:58.588628    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7" podUID="e0c034b6-0600-430b-aafa-f29fe15abb40" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:51:06.273571 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:06.273506    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7" podUID="e0c034b6-0600-430b-aafa-f29fe15abb40" containerName="main" containerID="cri-o://04f5a79a55d349480be083a35a8dc72981ed4a0cf76b10d3436ce893f3f35cf6" gracePeriod=45
Aug 04 18:51:06.522289 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:06.522256    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"
Aug 04 18:51:06.652432 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:06.652393    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc6pv\" (UniqueName: \"kubernetes.io/projected/e0c034b6-0600-430b-aafa-f29fe15abb40-kube-api-access-zc6pv\") pod \"e0c034b6-0600-430b-aafa-f29fe15abb40\" (UID: \"e0c034b6-0600-430b-aafa-f29fe15abb40\") "
Aug 04 18:51:06.652612 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:06.652444    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c034b6-0600-430b-aafa-f29fe15abb40-tls-certs\") pod \"e0c034b6-0600-430b-aafa-f29fe15abb40\" (UID: \"e0c034b6-0600-430b-aafa-f29fe15abb40\") "
Aug 04 18:51:06.654754 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:06.654720    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c034b6-0600-430b-aafa-f29fe15abb40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0c034b6-0600-430b-aafa-f29fe15abb40" (UID: "e0c034b6-0600-430b-aafa-f29fe15abb40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:51:06.654754 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:06.654733    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c034b6-0600-430b-aafa-f29fe15abb40-kube-api-access-zc6pv" (OuterVolumeSpecName: "kube-api-access-zc6pv") pod "e0c034b6-0600-430b-aafa-f29fe15abb40" (UID: "e0c034b6-0600-430b-aafa-f29fe15abb40"). InnerVolumeSpecName "kube-api-access-zc6pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:51:06.753469 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:06.753433    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc6pv\" (UniqueName: \"kubernetes.io/projected/e0c034b6-0600-430b-aafa-f29fe15abb40-kube-api-access-zc6pv\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:51:06.753469 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:06.753461    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c034b6-0600-430b-aafa-f29fe15abb40-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:51:07.238196 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:07.238154    2576 generic.go:356] "Generic (PLEG): container finished" podID="e0c034b6-0600-430b-aafa-f29fe15abb40" containerID="04f5a79a55d349480be083a35a8dc72981ed4a0cf76b10d3436ce893f3f35cf6" exitCode=0
Aug 04 18:51:07.238408 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:07.238203    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"
Aug 04 18:51:07.238408 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:07.238229    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7" event={"ID":"e0c034b6-0600-430b-aafa-f29fe15abb40","Type":"ContainerDied","Data":"04f5a79a55d349480be083a35a8dc72981ed4a0cf76b10d3436ce893f3f35cf6"}
Aug 04 18:51:07.238408 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:07.238271    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7" event={"ID":"e0c034b6-0600-430b-aafa-f29fe15abb40","Type":"ContainerDied","Data":"c8de8e77f7131d6de5cc1961863ede2d106d1b36f645c64dfb4d60684a89acb3"}
Aug 04 18:51:07.238408 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:07.238306    2576 scope.go:117] "RemoveContainer" containerID="04f5a79a55d349480be083a35a8dc72981ed4a0cf76b10d3436ce893f3f35cf6"
Aug 04 18:51:07.250168 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:07.250145    2576 scope.go:117] "RemoveContainer" containerID="04f5a79a55d349480be083a35a8dc72981ed4a0cf76b10d3436ce893f3f35cf6"
Aug 04 18:51:07.250499 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:51:07.250473    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f5a79a55d349480be083a35a8dc72981ed4a0cf76b10d3436ce893f3f35cf6\": container with ID starting with 04f5a79a55d349480be083a35a8dc72981ed4a0cf76b10d3436ce893f3f35cf6 not found: ID does not exist" containerID="04f5a79a55d349480be083a35a8dc72981ed4a0cf76b10d3436ce893f3f35cf6"
Aug 04 18:51:07.250552 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:07.250511    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f5a79a55d349480be083a35a8dc72981ed4a0cf76b10d3436ce893f3f35cf6"} err="failed to get container status \"04f5a79a55d349480be083a35a8dc72981ed4a0cf76b10d3436ce893f3f35cf6\": rpc error: code = NotFound desc = could not find container \"04f5a79a55d349480be083a35a8dc72981ed4a0cf76b10d3436ce893f3f35cf6\": container with ID starting with 04f5a79a55d349480be083a35a8dc72981ed4a0cf76b10d3436ce893f3f35cf6 not found: ID does not exist"
Aug 04 18:51:07.263827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:07.263799    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"]
Aug 04 18:51:07.267281 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:07.267257    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb6689h44m7"]
Aug 04 18:51:08.581673 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:08.581635    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0c034b6-0600-430b-aafa-f29fe15abb40" path="/var/lib/kubelet/pods/e0c034b6-0600-430b-aafa-f29fe15abb40/volumes"
Aug 04 18:51:11.331000 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.330966    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:11.424167 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.424133    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:11.543065 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.543028    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"]
Aug 04 18:51:11.543462 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.543444    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1714e8-afdf-4b1b-9595-d227bb488d84" containerName="main"
Aug 04 18:51:11.543462 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.543460    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1714e8-afdf-4b1b-9595-d227bb488d84" containerName="main"
Aug 04 18:51:11.543633 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.543475    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c034b6-0600-430b-aafa-f29fe15abb40" containerName="main"
Aug 04 18:51:11.543633 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.543481    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c034b6-0600-430b-aafa-f29fe15abb40" containerName="main"
Aug 04 18:51:11.543633 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.543554    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c1714e8-afdf-4b1b-9595-d227bb488d84" containerName="main"
Aug 04 18:51:11.543633 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.543570    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0c034b6-0600-430b-aafa-f29fe15abb40" containerName="main"
Aug 04 18:51:11.546650 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.546630    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.549646 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.549619    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-2mrx9\""
Aug 04 18:51:11.549889 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.549871    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Aug 04 18:51:11.549970 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.549928    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Aug 04 18:51:11.550243 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.550224    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Aug 04 18:51:11.560888 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.560866    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"]
Aug 04 18:51:11.698308 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.698214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c3581a-01ca-4600-af62-cc7dabd87db7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.698308 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.698258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.698308 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.698281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.698550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.698327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.698550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.698415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.698550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.698460    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.698550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.698477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.698550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.698506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwkgc\" (UniqueName: \"kubernetes.io/projected/90c3581a-01ca-4600-af62-cc7dabd87db7-kube-api-access-dwkgc\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.698550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.698533    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.800018 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.799982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c3581a-01ca-4600-af62-cc7dabd87db7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.800018 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.800029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.800334 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.800050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.800334 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.800220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.800334 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.800281    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.800334 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.800312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.800557 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.800334    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.800557 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.800376    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwkgc\" (UniqueName: \"kubernetes.io/projected/90c3581a-01ca-4600-af62-cc7dabd87db7-kube-api-access-dwkgc\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.800557 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.800418    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.800742 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.800553    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.800742 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.800575    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.800742 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.800641    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.800742 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.800693    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.800970 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.800945    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c3581a-01ca-4600-af62-cc7dabd87db7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.802648 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.802630    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.802987 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.802967    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.808124 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.808080    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwkgc\" (UniqueName: \"kubernetes.io/projected/90c3581a-01ca-4600-af62-cc7dabd87db7-kube-api-access-dwkgc\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.808459 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.808441    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-2s92w\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.857765 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.857729    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:11.932259 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.932225    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:11.999758 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:11.999728    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"]
Aug 04 18:51:12.002162 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:12.002130    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:12.002431 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:51:12.002408    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c3581a_01ca_4600_af62_cc7dabd87db7.slice/crio-3ac6b0b4ff25200fcc8611223075c4c3eb2d65a064575b41f9fa8394742ccdc6 WatchSource:0}: Error finding container 3ac6b0b4ff25200fcc8611223075c4c3eb2d65a064575b41f9fa8394742ccdc6: Status 404 returned error can't find the container with id 3ac6b0b4ff25200fcc8611223075c4c3eb2d65a064575b41f9fa8394742ccdc6
Aug 04 18:51:12.004539 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:12.004508    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:51:12.004605 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:12.004574    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:51:12.004658 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:12.004603    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:51:12.262746 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:12.262660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w" event={"ID":"90c3581a-01ca-4600-af62-cc7dabd87db7","Type":"ContainerStarted","Data":"47d9c2b012d8854d0e764a6b47acaadf3b8689098663729311253e952de1b2f6"}
Aug 04 18:51:12.262746 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:12.262717    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w" event={"ID":"90c3581a-01ca-4600-af62-cc7dabd87db7","Type":"ContainerStarted","Data":"3ac6b0b4ff25200fcc8611223075c4c3eb2d65a064575b41f9fa8394742ccdc6"}
Aug 04 18:51:12.285481 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:12.285422    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w" podStartSLOduration=1.2854015730000001 podStartE2EDuration="1.285401573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:51:12.282207817 +0000 UTC m=+3072.258039514" watchObservedRunningTime="2026-08-04 18:51:12.285401573 +0000 UTC m=+3072.261233271"
Aug 04 18:51:12.858390 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:12.858354    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:13.862918 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:13.862885    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:14.273241 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:14.273141    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:14.274158 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:14.274138    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:15.265094 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:15.265002    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:18.141991 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:18.141956    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:20.704589 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:20.704533    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:20.908327 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:20.908293    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"]
Aug 04 18:51:20.911786 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:20.911765    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"
Aug 04 18:51:20.915307 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:20.915283    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-wklt5\""
Aug 04 18:51:20.915468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:20.915344    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-ssdb9\""
Aug 04 18:51:20.915468 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:20.915393    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Aug 04 18:51:20.922588 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:20.922565    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"]
Aug 04 18:51:20.983359 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:20.983265    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2676ee55-f684-4068-855d-56c9bdda5660-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn\" (UID: \"2676ee55-f684-4068-855d-56c9bdda5660\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"
Aug 04 18:51:20.983530 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:20.983396    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgjgs\" (UniqueName: \"kubernetes.io/projected/2676ee55-f684-4068-855d-56c9bdda5660-kube-api-access-vgjgs\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn\" (UID: \"2676ee55-f684-4068-855d-56c9bdda5660\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"
Aug 04 18:51:21.005537 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:21.005472    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:21.084139 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:21.084072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2676ee55-f684-4068-855d-56c9bdda5660-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn\" (UID: \"2676ee55-f684-4068-855d-56c9bdda5660\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"
Aug 04 18:51:21.084331 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:21.084185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgjgs\" (UniqueName: \"kubernetes.io/projected/2676ee55-f684-4068-855d-56c9bdda5660-kube-api-access-vgjgs\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn\" (UID: \"2676ee55-f684-4068-855d-56c9bdda5660\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"
Aug 04 18:51:21.086749 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:21.086720    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2676ee55-f684-4068-855d-56c9bdda5660-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn\" (UID: \"2676ee55-f684-4068-855d-56c9bdda5660\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"
Aug 04 18:51:21.092951 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:21.092919    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgjgs\" (UniqueName: \"kubernetes.io/projected/2676ee55-f684-4068-855d-56c9bdda5660-kube-api-access-vgjgs\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn\" (UID: \"2676ee55-f684-4068-855d-56c9bdda5660\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"
Aug 04 18:51:21.103819 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:21.103791    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:21.223529 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:21.223492    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"
Aug 04 18:51:21.387301 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:21.387268    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"]
Aug 04 18:51:21.391622 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:51:21.391591    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2676ee55_f684_4068_855d_56c9bdda5660.slice/crio-934e0704bd99d2f8aeac29b982c02d94c04b9fd66526dd3a9c747ea9892ca494 WatchSource:0}: Error finding container 934e0704bd99d2f8aeac29b982c02d94c04b9fd66526dd3a9c747ea9892ca494: Status 404 returned error can't find the container with id 934e0704bd99d2f8aeac29b982c02d94c04b9fd66526dd3a9c747ea9892ca494
Aug 04 18:51:22.316514 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:22.316475    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn" event={"ID":"2676ee55-f684-4068-855d-56c9bdda5660","Type":"ContainerStarted","Data":"884136ef147804c85c417d46b73b6de2c1ce16864a3d76c5abccc1e833a73a04"}
Aug 04 18:51:22.317003 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:22.316532    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn" event={"ID":"2676ee55-f684-4068-855d-56c9bdda5660","Type":"ContainerStarted","Data":"934e0704bd99d2f8aeac29b982c02d94c04b9fd66526dd3a9c747ea9892ca494"}
Aug 04 18:51:22.317003 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:22.316612    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"
Aug 04 18:51:22.333827 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:22.333778    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn" podStartSLOduration=2.33376373 podStartE2EDuration="2.33376373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:51:22.332469843 +0000 UTC m=+3082.308301565" watchObservedRunningTime="2026-08-04 18:51:22.33376373 +0000 UTC m=+3082.309595427"
Aug 04 18:51:24.060475 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:24.060425    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:24.123384 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:24.123346    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:24.326896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:24.326805    2576 generic.go:356] "Generic (PLEG): container finished" podID="90c3581a-01ca-4600-af62-cc7dabd87db7" containerID="47d9c2b012d8854d0e764a6b47acaadf3b8689098663729311253e952de1b2f6" exitCode=0
Aug 04 18:51:24.326896 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:24.326873    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w" event={"ID":"90c3581a-01ca-4600-af62-cc7dabd87db7","Type":"ContainerDied","Data":"47d9c2b012d8854d0e764a6b47acaadf3b8689098663729311253e952de1b2f6"}
Aug 04 18:51:24.327304 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:24.327284    2576 scope.go:117] "RemoveContainer" containerID="47d9c2b012d8854d0e764a6b47acaadf3b8689098663729311253e952de1b2f6"
Aug 04 18:51:24.328191 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:24.328168    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:51:24.328252 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:24.328239    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:51:24.328289 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:24.328270    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Aug 04 18:51:25.332895 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:25.332861    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w" event={"ID":"90c3581a-01ca-4600-af62-cc7dabd87db7","Type":"ContainerStarted","Data":"79cef72eb21f18bf2687cf81e8f963b6cbf3cbdb75c463a60c9e846a2326ebc2"}
Aug 04 18:51:25.858937 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:25.858888    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:25.860434 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:25.860388    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w" podUID="90c3581a-01ca-4600-af62-cc7dabd87db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Aug 04 18:51:26.858891 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:26.858843    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w" podUID="90c3581a-01ca-4600-af62-cc7dabd87db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Aug 04 18:51:27.858412 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:27.858358    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w" podUID="90c3581a-01ca-4600-af62-cc7dabd87db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Aug 04 18:51:28.858416 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:28.858363    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w" podUID="90c3581a-01ca-4600-af62-cc7dabd87db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Aug 04 18:51:29.273653 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:29.273560    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:29.858634 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:29.858578    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w" podUID="90c3581a-01ca-4600-af62-cc7dabd87db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Aug 04 18:51:30.515364 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:30.515333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:30.862537 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:30.862509    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:30.863528 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:30.863508    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:51:50.515147 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:50.515076    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:50.615290 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:50.615252    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:50.715226 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:50.715184    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:53.322798 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:53.322763    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"
Aug 04 18:51:55.552156 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:55.552086    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:55.893007 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:55.892917    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:57.506595 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:57.506554    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:57.622723 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:51:57.622689    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:52:11.024464 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:52:11.024425    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:52:50.966227 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:52:50.966186    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:25.387576 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:25.387540    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"]
Aug 04 18:53:26.113074 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:26.113038    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:27.722926 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:27.722892    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:33.322760 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:33.322712    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn" podUID="2676ee55-f684-4068-855d-56c9bdda5660" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:53:33.386874 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:33.386837    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:33.488130 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:33.488085    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:40.406848 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:40.406790    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn" podUID="2676ee55-f684-4068-855d-56c9bdda5660" containerName="main" containerID="cri-o://884136ef147804c85c417d46b73b6de2c1ce16864a3d76c5abccc1e833a73a04" gracePeriod=45
Aug 04 18:53:40.662823 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:40.662760    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"
Aug 04 18:53:40.759199 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:40.759163    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgjgs\" (UniqueName: \"kubernetes.io/projected/2676ee55-f684-4068-855d-56c9bdda5660-kube-api-access-vgjgs\") pod \"2676ee55-f684-4068-855d-56c9bdda5660\" (UID: \"2676ee55-f684-4068-855d-56c9bdda5660\") "
Aug 04 18:53:40.759395 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:40.759304    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2676ee55-f684-4068-855d-56c9bdda5660-tls-certs\") pod \"2676ee55-f684-4068-855d-56c9bdda5660\" (UID: \"2676ee55-f684-4068-855d-56c9bdda5660\") "
Aug 04 18:53:40.761454 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:40.761412    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2676ee55-f684-4068-855d-56c9bdda5660-kube-api-access-vgjgs" (OuterVolumeSpecName: "kube-api-access-vgjgs") pod "2676ee55-f684-4068-855d-56c9bdda5660" (UID: "2676ee55-f684-4068-855d-56c9bdda5660"). InnerVolumeSpecName "kube-api-access-vgjgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:53:40.761454 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:40.761435    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2676ee55-f684-4068-855d-56c9bdda5660-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2676ee55-f684-4068-855d-56c9bdda5660" (UID: "2676ee55-f684-4068-855d-56c9bdda5660"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:53:40.860018 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:40.859981    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgjgs\" (UniqueName: \"kubernetes.io/projected/2676ee55-f684-4068-855d-56c9bdda5660-kube-api-access-vgjgs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:53:40.860018 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:40.860020    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2676ee55-f684-4068-855d-56c9bdda5660-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:53:41.005160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:41.005062    2576 generic.go:356] "Generic (PLEG): container finished" podID="2676ee55-f684-4068-855d-56c9bdda5660" containerID="884136ef147804c85c417d46b73b6de2c1ce16864a3d76c5abccc1e833a73a04" exitCode=0
Aug 04 18:53:41.005160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:41.005120    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"
Aug 04 18:53:41.005346 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:41.005153    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn" event={"ID":"2676ee55-f684-4068-855d-56c9bdda5660","Type":"ContainerDied","Data":"884136ef147804c85c417d46b73b6de2c1ce16864a3d76c5abccc1e833a73a04"}
Aug 04 18:53:41.005346 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:41.005209    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn" event={"ID":"2676ee55-f684-4068-855d-56c9bdda5660","Type":"ContainerDied","Data":"934e0704bd99d2f8aeac29b982c02d94c04b9fd66526dd3a9c747ea9892ca494"}
Aug 04 18:53:41.005346 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:41.005233    2576 scope.go:117] "RemoveContainer" containerID="884136ef147804c85c417d46b73b6de2c1ce16864a3d76c5abccc1e833a73a04"
Aug 04 18:53:41.014797 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:41.014780    2576 scope.go:117] "RemoveContainer" containerID="884136ef147804c85c417d46b73b6de2c1ce16864a3d76c5abccc1e833a73a04"
Aug 04 18:53:41.015064 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:53:41.015043    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884136ef147804c85c417d46b73b6de2c1ce16864a3d76c5abccc1e833a73a04\": container with ID starting with 884136ef147804c85c417d46b73b6de2c1ce16864a3d76c5abccc1e833a73a04 not found: ID does not exist" containerID="884136ef147804c85c417d46b73b6de2c1ce16864a3d76c5abccc1e833a73a04"
Aug 04 18:53:41.015162 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:41.015072    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884136ef147804c85c417d46b73b6de2c1ce16864a3d76c5abccc1e833a73a04"} err="failed to get container status \"884136ef147804c85c417d46b73b6de2c1ce16864a3d76c5abccc1e833a73a04\": rpc error: code = NotFound desc = could not find container \"884136ef147804c85c417d46b73b6de2c1ce16864a3d76c5abccc1e833a73a04\": container with ID starting with 884136ef147804c85c417d46b73b6de2c1ce16864a3d76c5abccc1e833a73a04 not found: ID does not exist"
Aug 04 18:53:41.030222 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:41.030196    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"]
Aug 04 18:53:41.034999 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:41.034975    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-xknhn"]
Aug 04 18:53:42.581925 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:42.581892    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2676ee55-f684-4068-855d-56c9bdda5660" path="/var/lib/kubelet/pods/2676ee55-f684-4068-855d-56c9bdda5660/volumes"
Aug 04 18:53:44.835721 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:44.835684    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:45.258792 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:45.258711    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:45.349791 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:45.349710    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:50.207414 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:50.207377    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"]
Aug 04 18:53:50.207888 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:50.207584    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w" podUID="90c3581a-01ca-4600-af62-cc7dabd87db7" containerName="istio-proxy" containerID="cri-o://79cef72eb21f18bf2687cf81e8f963b6cbf3cbdb75c463a60c9e846a2326ebc2" gracePeriod=30
Aug 04 18:53:50.295657 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:50.295620    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:55.452044 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.452015    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:53:55.496965 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.496884    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-envoy\") pod \"90c3581a-01ca-4600-af62-cc7dabd87db7\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") "
Aug 04 18:53:55.496965 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.496926    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-data\") pod \"90c3581a-01ca-4600-af62-cc7dabd87db7\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") "
Aug 04 18:53:55.496965 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.496957    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-workload-certs\") pod \"90c3581a-01ca-4600-af62-cc7dabd87db7\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") "
Aug 04 18:53:55.497305 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.496985    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwkgc\" (UniqueName: \"kubernetes.io/projected/90c3581a-01ca-4600-af62-cc7dabd87db7-kube-api-access-dwkgc\") pod \"90c3581a-01ca-4600-af62-cc7dabd87db7\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") "
Aug 04 18:53:55.497305 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.497013    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c3581a-01ca-4600-af62-cc7dabd87db7-istiod-ca-cert\") pod \"90c3581a-01ca-4600-af62-cc7dabd87db7\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") "
Aug 04 18:53:55.497305 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.497047    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-workload-socket\") pod \"90c3581a-01ca-4600-af62-cc7dabd87db7\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") "
Aug 04 18:53:55.497305 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.497266    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "90c3581a-01ca-4600-af62-cc7dabd87db7" (UID: "90c3581a-01ca-4600-af62-cc7dabd87db7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:53:55.497527 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.497327    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-data" (OuterVolumeSpecName: "istio-data") pod "90c3581a-01ca-4600-af62-cc7dabd87db7" (UID: "90c3581a-01ca-4600-af62-cc7dabd87db7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:53:55.497527 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.497370    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-credential-socket\") pod \"90c3581a-01ca-4600-af62-cc7dabd87db7\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") "
Aug 04 18:53:55.497527 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.497466    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-token\") pod \"90c3581a-01ca-4600-af62-cc7dabd87db7\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") "
Aug 04 18:53:55.497527 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.497481    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "90c3581a-01ca-4600-af62-cc7dabd87db7" (UID: "90c3581a-01ca-4600-af62-cc7dabd87db7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:53:55.497527 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.497503    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-podinfo\") pod \"90c3581a-01ca-4600-af62-cc7dabd87db7\" (UID: \"90c3581a-01ca-4600-af62-cc7dabd87db7\") "
Aug 04 18:53:55.497527 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.497517    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c3581a-01ca-4600-af62-cc7dabd87db7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "90c3581a-01ca-4600-af62-cc7dabd87db7" (UID: "90c3581a-01ca-4600-af62-cc7dabd87db7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:53:55.497816 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.497639    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "90c3581a-01ca-4600-af62-cc7dabd87db7" (UID: "90c3581a-01ca-4600-af62-cc7dabd87db7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:53:55.498033 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.497953    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:53:55.498033 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.497979    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:53:55.498033 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.497994    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c3581a-01ca-4600-af62-cc7dabd87db7-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:53:55.498033 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.498007    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:53:55.498033 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.498022    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:53:55.499817 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.499765    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c3581a-01ca-4600-af62-cc7dabd87db7-kube-api-access-dwkgc" (OuterVolumeSpecName: "kube-api-access-dwkgc") pod "90c3581a-01ca-4600-af62-cc7dabd87db7" (UID: "90c3581a-01ca-4600-af62-cc7dabd87db7"). InnerVolumeSpecName "kube-api-access-dwkgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:53:55.500021 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.499994    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-token" (OuterVolumeSpecName: "istio-token") pod "90c3581a-01ca-4600-af62-cc7dabd87db7" (UID: "90c3581a-01ca-4600-af62-cc7dabd87db7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:53:55.500172 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.500151    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "90c3581a-01ca-4600-af62-cc7dabd87db7" (UID: "90c3581a-01ca-4600-af62-cc7dabd87db7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:53:55.500266 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.500173    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "90c3581a-01ca-4600-af62-cc7dabd87db7" (UID: "90c3581a-01ca-4600-af62-cc7dabd87db7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 04 18:53:55.598469 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.598440    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwkgc\" (UniqueName: \"kubernetes.io/projected/90c3581a-01ca-4600-af62-cc7dabd87db7-kube-api-access-dwkgc\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:53:55.598469 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.598466    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:53:55.598657 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.598477    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:53:55.598657 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:55.598490    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c3581a-01ca-4600-af62-cc7dabd87db7-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:53:56.080962 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:56.080928    2576 generic.go:356] "Generic (PLEG): container finished" podID="90c3581a-01ca-4600-af62-cc7dabd87db7" containerID="79cef72eb21f18bf2687cf81e8f963b6cbf3cbdb75c463a60c9e846a2326ebc2" exitCode=0
Aug 04 18:53:56.081185 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:56.080968    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w" event={"ID":"90c3581a-01ca-4600-af62-cc7dabd87db7","Type":"ContainerDied","Data":"79cef72eb21f18bf2687cf81e8f963b6cbf3cbdb75c463a60c9e846a2326ebc2"}
Aug 04 18:53:56.081185 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:56.080974    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"
Aug 04 18:53:56.081185 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:56.081006    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w" event={"ID":"90c3581a-01ca-4600-af62-cc7dabd87db7","Type":"ContainerDied","Data":"3ac6b0b4ff25200fcc8611223075c4c3eb2d65a064575b41f9fa8394742ccdc6"}
Aug 04 18:53:56.081185 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:56.081027    2576 scope.go:117] "RemoveContainer" containerID="79cef72eb21f18bf2687cf81e8f963b6cbf3cbdb75c463a60c9e846a2326ebc2"
Aug 04 18:53:56.090098 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:56.090074    2576 scope.go:117] "RemoveContainer" containerID="47d9c2b012d8854d0e764a6b47acaadf3b8689098663729311253e952de1b2f6"
Aug 04 18:53:56.097868 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:56.097851    2576 scope.go:117] "RemoveContainer" containerID="79cef72eb21f18bf2687cf81e8f963b6cbf3cbdb75c463a60c9e846a2326ebc2"
Aug 04 18:53:56.098095 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:53:56.098079    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79cef72eb21f18bf2687cf81e8f963b6cbf3cbdb75c463a60c9e846a2326ebc2\": container with ID starting with 79cef72eb21f18bf2687cf81e8f963b6cbf3cbdb75c463a60c9e846a2326ebc2 not found: ID does not exist" containerID="79cef72eb21f18bf2687cf81e8f963b6cbf3cbdb75c463a60c9e846a2326ebc2"
Aug 04 18:53:56.098160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:56.098104    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79cef72eb21f18bf2687cf81e8f963b6cbf3cbdb75c463a60c9e846a2326ebc2"} err="failed to get container status \"79cef72eb21f18bf2687cf81e8f963b6cbf3cbdb75c463a60c9e846a2326ebc2\": rpc error: code = NotFound desc = could not find container \"79cef72eb21f18bf2687cf81e8f963b6cbf3cbdb75c463a60c9e846a2326ebc2\": container with ID starting with 79cef72eb21f18bf2687cf81e8f963b6cbf3cbdb75c463a60c9e846a2326ebc2 not found: ID does not exist"
Aug 04 18:53:56.098160 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:56.098143    2576 scope.go:117] "RemoveContainer" containerID="47d9c2b012d8854d0e764a6b47acaadf3b8689098663729311253e952de1b2f6"
Aug 04 18:53:56.098398 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:53:56.098374    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d9c2b012d8854d0e764a6b47acaadf3b8689098663729311253e952de1b2f6\": container with ID starting with 47d9c2b012d8854d0e764a6b47acaadf3b8689098663729311253e952de1b2f6 not found: ID does not exist" containerID="47d9c2b012d8854d0e764a6b47acaadf3b8689098663729311253e952de1b2f6"
Aug 04 18:53:56.098467 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:56.098408    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d9c2b012d8854d0e764a6b47acaadf3b8689098663729311253e952de1b2f6"} err="failed to get container status \"47d9c2b012d8854d0e764a6b47acaadf3b8689098663729311253e952de1b2f6\": rpc error: code = NotFound desc = could not find container \"47d9c2b012d8854d0e764a6b47acaadf3b8689098663729311253e952de1b2f6\": container with ID starting with 47d9c2b012d8854d0e764a6b47acaadf3b8689098663729311253e952de1b2f6 not found: ID does not exist"
Aug 04 18:53:56.105300 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:56.105276    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"]
Aug 04 18:53:56.108813 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:56.108789    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-2s92w"]
Aug 04 18:53:56.582614 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:53:56.582580    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90c3581a-01ca-4600-af62-cc7dabd87db7" path="/var/lib/kubelet/pods/90c3581a-01ca-4600-af62-cc7dabd87db7/volumes"
Aug 04 18:54:05.246615 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:05.246583    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:05.340982 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:05.340940    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:11.256299 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.256255    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:11.487890 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.487850    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"]
Aug 04 18:54:11.488305 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.488288    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c3581a-01ca-4600-af62-cc7dabd87db7" containerName="istio-proxy"
Aug 04 18:54:11.488365 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.488308    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c3581a-01ca-4600-af62-cc7dabd87db7" containerName="istio-proxy"
Aug 04 18:54:11.488365 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.488319    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c3581a-01ca-4600-af62-cc7dabd87db7" containerName="istio-proxy"
Aug 04 18:54:11.488365 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.488325    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c3581a-01ca-4600-af62-cc7dabd87db7" containerName="istio-proxy"
Aug 04 18:54:11.488365 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.488348    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2676ee55-f684-4068-855d-56c9bdda5660" containerName="main"
Aug 04 18:54:11.488365 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.488354    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2676ee55-f684-4068-855d-56c9bdda5660" containerName="main"
Aug 04 18:54:11.488520 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.488415    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="90c3581a-01ca-4600-af62-cc7dabd87db7" containerName="istio-proxy"
Aug 04 18:54:11.488520 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.488423    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2676ee55-f684-4068-855d-56c9bdda5660" containerName="main"
Aug 04 18:54:11.493043 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.493020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"
Aug 04 18:54:11.496816 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.496785    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-czcbj\""
Aug 04 18:54:11.496949 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.496822    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Aug 04 18:54:11.496949 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.496785    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Aug 04 18:54:11.496949 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.496786    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Aug 04 18:54:11.496949 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.496786    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-8wdmq\""
Aug 04 18:54:11.501801 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.501779    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"]
Aug 04 18:54:11.647213 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.647167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d64017e-bb81-409a-bb6c-ede0393455ca-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8\" (UID: \"3d64017e-bb81-409a-bb6c-ede0393455ca\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"
Aug 04 18:54:11.647479 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.647249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qm2\" (UniqueName: \"kubernetes.io/projected/3d64017e-bb81-409a-bb6c-ede0393455ca-kube-api-access-f4qm2\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8\" (UID: \"3d64017e-bb81-409a-bb6c-ede0393455ca\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"
Aug 04 18:54:11.748265 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.748231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qm2\" (UniqueName: \"kubernetes.io/projected/3d64017e-bb81-409a-bb6c-ede0393455ca-kube-api-access-f4qm2\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8\" (UID: \"3d64017e-bb81-409a-bb6c-ede0393455ca\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"
Aug 04 18:54:11.748448 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.748305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d64017e-bb81-409a-bb6c-ede0393455ca-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8\" (UID: \"3d64017e-bb81-409a-bb6c-ede0393455ca\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"
Aug 04 18:54:11.751225 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.751182    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d64017e-bb81-409a-bb6c-ede0393455ca-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8\" (UID: \"3d64017e-bb81-409a-bb6c-ede0393455ca\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"
Aug 04 18:54:11.760991 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.760951    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qm2\" (UniqueName: \"kubernetes.io/projected/3d64017e-bb81-409a-bb6c-ede0393455ca-kube-api-access-f4qm2\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8\" (UID: \"3d64017e-bb81-409a-bb6c-ede0393455ca\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"
Aug 04 18:54:11.792410 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.792379    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:11.805255 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.805231    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"
Aug 04 18:54:11.945447 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:11.945355    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"]
Aug 04 18:54:11.947879 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:54:11.947837    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d64017e_bb81_409a_bb6c_ede0393455ca.slice/crio-1f45916a81401ca7c6927d3b050b241b3042255b863a357cf5c99e645a375161 WatchSource:0}: Error finding container 1f45916a81401ca7c6927d3b050b241b3042255b863a357cf5c99e645a375161: Status 404 returned error can't find the container with id 1f45916a81401ca7c6927d3b050b241b3042255b863a357cf5c99e645a375161
Aug 04 18:54:12.034653 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:12.034616    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:12.134781 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:12.134747    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:12.161090 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:12.161054    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8" event={"ID":"3d64017e-bb81-409a-bb6c-ede0393455ca","Type":"ContainerStarted","Data":"f110bc07518500001d2733c58c90b5a8daed20a1159beb94ff8d797bfc30efdd"}
Aug 04 18:54:12.161272 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:12.161120    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8" event={"ID":"3d64017e-bb81-409a-bb6c-ede0393455ca","Type":"ContainerStarted","Data":"1f45916a81401ca7c6927d3b050b241b3042255b863a357cf5c99e645a375161"}
Aug 04 18:54:12.161272 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:12.161190    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"
Aug 04 18:54:12.180003 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:12.179950    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8" podStartSLOduration=1.17993245 podStartE2EDuration="1.17993245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:54:12.178267544 +0000 UTC m=+3252.154099242" watchObservedRunningTime="2026-08-04 18:54:12.17993245 +0000 UTC m=+3252.155764147"
Aug 04 18:54:12.264487 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:12.264372    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:12.707208 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:12.707161    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:18.479574 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:18.479535    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:21.150932 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.150890    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:21.342018 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.341983    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"]
Aug 04 18:54:21.342457 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.342442    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="90c3581a-01ca-4600-af62-cc7dabd87db7" containerName="istio-proxy"
Aug 04 18:54:21.345301 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.345283    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"
Aug 04 18:54:21.349129 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.349089    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Aug 04 18:54:21.349268 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.349145    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-qgkj2\""
Aug 04 18:54:21.349268 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.349145    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Aug 04 18:54:21.349393 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.349146    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-dwscp\""
Aug 04 18:54:21.349393 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.349158    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Aug 04 18:54:21.354978 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.354953    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"]
Aug 04 18:54:21.370475 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.370424    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:21.432596 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.432491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd6d54d-0ff7-4da0-b03e-4ffd9efc3505-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-xpp25\" (UID: \"efd6d54d-0ff7-4da0-b03e-4ffd9efc3505\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"
Aug 04 18:54:21.432596 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.432559    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb46d\" (UniqueName: \"kubernetes.io/projected/efd6d54d-0ff7-4da0-b03e-4ffd9efc3505-kube-api-access-gb46d\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-xpp25\" (UID: \"efd6d54d-0ff7-4da0-b03e-4ffd9efc3505\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"
Aug 04 18:54:21.533426 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.533384    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd6d54d-0ff7-4da0-b03e-4ffd9efc3505-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-xpp25\" (UID: \"efd6d54d-0ff7-4da0-b03e-4ffd9efc3505\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"
Aug 04 18:54:21.533625 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.533457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb46d\" (UniqueName: \"kubernetes.io/projected/efd6d54d-0ff7-4da0-b03e-4ffd9efc3505-kube-api-access-gb46d\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-xpp25\" (UID: \"efd6d54d-0ff7-4da0-b03e-4ffd9efc3505\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"
Aug 04 18:54:21.535913 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.535882    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd6d54d-0ff7-4da0-b03e-4ffd9efc3505-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-xpp25\" (UID: \"efd6d54d-0ff7-4da0-b03e-4ffd9efc3505\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"
Aug 04 18:54:21.543383 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.543356    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb46d\" (UniqueName: \"kubernetes.io/projected/efd6d54d-0ff7-4da0-b03e-4ffd9efc3505-kube-api-access-gb46d\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-xpp25\" (UID: \"efd6d54d-0ff7-4da0-b03e-4ffd9efc3505\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"
Aug 04 18:54:21.657539 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.657501    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"
Aug 04 18:54:21.795139 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:21.795088    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"]
Aug 04 18:54:21.797525 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:54:21.797500    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd6d54d_0ff7_4da0_b03e_4ffd9efc3505.slice/crio-921e1462b68b81c2a092ee69ac37968526c16667b3d839c77f5a31e385e5fb02 WatchSource:0}: Error finding container 921e1462b68b81c2a092ee69ac37968526c16667b3d839c77f5a31e385e5fb02: Status 404 returned error can't find the container with id 921e1462b68b81c2a092ee69ac37968526c16667b3d839c77f5a31e385e5fb02
Aug 04 18:54:22.214129 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:22.214061    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25" event={"ID":"efd6d54d-0ff7-4da0-b03e-4ffd9efc3505","Type":"ContainerStarted","Data":"01b0232f900eb0a690e7b74eb89e8e996336bf962ee830a7f70af74ae5bbcd6d"}
Aug 04 18:54:22.214613 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:22.214152    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25" event={"ID":"efd6d54d-0ff7-4da0-b03e-4ffd9efc3505","Type":"ContainerStarted","Data":"921e1462b68b81c2a092ee69ac37968526c16667b3d839c77f5a31e385e5fb02"}
Aug 04 18:54:22.214613 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:22.214186    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"
Aug 04 18:54:22.233674 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:22.233619    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25" podStartSLOduration=1.233598721 podStartE2EDuration="1.233598721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:54:22.233452513 +0000 UTC m=+3262.209284212" watchObservedRunningTime="2026-08-04 18:54:22.233598721 +0000 UTC m=+3262.209430419"
Aug 04 18:54:23.976133 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:23.976084    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:24.404795 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:24.404755    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:24.505346 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:24.505316    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:28.451561 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:28.451512    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:42.750834 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:42.750795    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:43.167470 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:43.167440    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"
Aug 04 18:54:49.418913 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:49.418877    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:53.221054 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:54:53.221022    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"
Aug 04 18:56:50.453354 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:56:50.453310    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"]
Aug 04 18:56:50.479421 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:56:50.479384    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:56:50.541970 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:56:50.541928    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:56:53.167821 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:56:53.167734    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8" podUID="3d64017e-bb81-409a-bb6c-ede0393455ca" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:56:56.442919 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:56:56.442879    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"]
Aug 04 18:56:56.511901 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:56:56.511872    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:56:56.573042 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:56:56.572990    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:03.167226 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:03.167180    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8" podUID="3d64017e-bb81-409a-bb6c-ede0393455ca" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:57:03.221642 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:03.221601    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25" podUID="efd6d54d-0ff7-4da0-b03e-4ffd9efc3505" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:57:05.473716 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:05.473675    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8" podUID="3d64017e-bb81-409a-bb6c-ede0393455ca" containerName="main" containerID="cri-o://f110bc07518500001d2733c58c90b5a8daed20a1159beb94ff8d797bfc30efdd" gracePeriod=45
Aug 04 18:57:05.727655 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:05.727595    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"
Aug 04 18:57:05.830489 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:05.830452    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4qm2\" (UniqueName: \"kubernetes.io/projected/3d64017e-bb81-409a-bb6c-ede0393455ca-kube-api-access-f4qm2\") pod \"3d64017e-bb81-409a-bb6c-ede0393455ca\" (UID: \"3d64017e-bb81-409a-bb6c-ede0393455ca\") "
Aug 04 18:57:05.830686 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:05.830531    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d64017e-bb81-409a-bb6c-ede0393455ca-tls-certs\") pod \"3d64017e-bb81-409a-bb6c-ede0393455ca\" (UID: \"3d64017e-bb81-409a-bb6c-ede0393455ca\") "
Aug 04 18:57:05.832742 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:05.832712    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d64017e-bb81-409a-bb6c-ede0393455ca-kube-api-access-f4qm2" (OuterVolumeSpecName: "kube-api-access-f4qm2") pod "3d64017e-bb81-409a-bb6c-ede0393455ca" (UID: "3d64017e-bb81-409a-bb6c-ede0393455ca"). InnerVolumeSpecName "kube-api-access-f4qm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:57:05.832842 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:05.832714    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d64017e-bb81-409a-bb6c-ede0393455ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d64017e-bb81-409a-bb6c-ede0393455ca" (UID: "3d64017e-bb81-409a-bb6c-ede0393455ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:57:05.931584 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:05.931546    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4qm2\" (UniqueName: \"kubernetes.io/projected/3d64017e-bb81-409a-bb6c-ede0393455ca-kube-api-access-f4qm2\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:57:05.931584 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:05.931576    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d64017e-bb81-409a-bb6c-ede0393455ca-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.008183 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:06.008072    2576 generic.go:356] "Generic (PLEG): container finished" podID="3d64017e-bb81-409a-bb6c-ede0393455ca" containerID="f110bc07518500001d2733c58c90b5a8daed20a1159beb94ff8d797bfc30efdd" exitCode=0
Aug 04 18:57:06.008183 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:06.008159    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8" event={"ID":"3d64017e-bb81-409a-bb6c-ede0393455ca","Type":"ContainerDied","Data":"f110bc07518500001d2733c58c90b5a8daed20a1159beb94ff8d797bfc30efdd"}
Aug 04 18:57:06.008371 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:06.008236    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8" event={"ID":"3d64017e-bb81-409a-bb6c-ede0393455ca","Type":"ContainerDied","Data":"1f45916a81401ca7c6927d3b050b241b3042255b863a357cf5c99e645a375161"}
Aug 04 18:57:06.008371 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:06.008172    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"
Aug 04 18:57:06.008371 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:06.008258    2576 scope.go:117] "RemoveContainer" containerID="f110bc07518500001d2733c58c90b5a8daed20a1159beb94ff8d797bfc30efdd"
Aug 04 18:57:06.018222 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:06.018190    2576 scope.go:117] "RemoveContainer" containerID="f110bc07518500001d2733c58c90b5a8daed20a1159beb94ff8d797bfc30efdd"
Aug 04 18:57:06.018494 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:57:06.018474    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f110bc07518500001d2733c58c90b5a8daed20a1159beb94ff8d797bfc30efdd\": container with ID starting with f110bc07518500001d2733c58c90b5a8daed20a1159beb94ff8d797bfc30efdd not found: ID does not exist" containerID="f110bc07518500001d2733c58c90b5a8daed20a1159beb94ff8d797bfc30efdd"
Aug 04 18:57:06.018551 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:06.018502    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f110bc07518500001d2733c58c90b5a8daed20a1159beb94ff8d797bfc30efdd"} err="failed to get container status \"f110bc07518500001d2733c58c90b5a8daed20a1159beb94ff8d797bfc30efdd\": rpc error: code = NotFound desc = could not find container \"f110bc07518500001d2733c58c90b5a8daed20a1159beb94ff8d797bfc30efdd\": container with ID starting with f110bc07518500001d2733c58c90b5a8daed20a1159beb94ff8d797bfc30efdd not found: ID does not exist"
Aug 04 18:57:06.031381 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:06.031352    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"]
Aug 04 18:57:06.033515 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:06.033493    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5nxh8"]
Aug 04 18:57:06.582052 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:06.582015    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d64017e-bb81-409a-bb6c-ede0393455ca" path="/var/lib/kubelet/pods/3d64017e-bb81-409a-bb6c-ede0393455ca/volumes"
Aug 04 18:57:10.482385 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:10.482351    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:10.582179 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:10.582136    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:11.469297 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:11.469246    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25" podUID="efd6d54d-0ff7-4da0-b03e-4ffd9efc3505" containerName="main" containerID="cri-o://01b0232f900eb0a690e7b74eb89e8e996336bf962ee830a7f70af74ae5bbcd6d" gracePeriod=45
Aug 04 18:57:11.718627 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:11.718600    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"
Aug 04 18:57:11.885798 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:11.885759    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb46d\" (UniqueName: \"kubernetes.io/projected/efd6d54d-0ff7-4da0-b03e-4ffd9efc3505-kube-api-access-gb46d\") pod \"efd6d54d-0ff7-4da0-b03e-4ffd9efc3505\" (UID: \"efd6d54d-0ff7-4da0-b03e-4ffd9efc3505\") "
Aug 04 18:57:11.886011 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:11.885836    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd6d54d-0ff7-4da0-b03e-4ffd9efc3505-tls-certs\") pod \"efd6d54d-0ff7-4da0-b03e-4ffd9efc3505\" (UID: \"efd6d54d-0ff7-4da0-b03e-4ffd9efc3505\") "
Aug 04 18:57:11.888167 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:11.888127    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd6d54d-0ff7-4da0-b03e-4ffd9efc3505-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efd6d54d-0ff7-4da0-b03e-4ffd9efc3505" (UID: "efd6d54d-0ff7-4da0-b03e-4ffd9efc3505"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:57:11.888293 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:11.888162    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd6d54d-0ff7-4da0-b03e-4ffd9efc3505-kube-api-access-gb46d" (OuterVolumeSpecName: "kube-api-access-gb46d") pod "efd6d54d-0ff7-4da0-b03e-4ffd9efc3505" (UID: "efd6d54d-0ff7-4da0-b03e-4ffd9efc3505"). InnerVolumeSpecName "kube-api-access-gb46d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:57:11.986477 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:11.986442    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb46d\" (UniqueName: \"kubernetes.io/projected/efd6d54d-0ff7-4da0-b03e-4ffd9efc3505-kube-api-access-gb46d\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:57:11.986477 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:11.986470    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd6d54d-0ff7-4da0-b03e-4ffd9efc3505-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 18:57:12.041295 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:12.041254    2576 generic.go:356] "Generic (PLEG): container finished" podID="efd6d54d-0ff7-4da0-b03e-4ffd9efc3505" containerID="01b0232f900eb0a690e7b74eb89e8e996336bf962ee830a7f70af74ae5bbcd6d" exitCode=0
Aug 04 18:57:12.041482 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:12.041302    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"
Aug 04 18:57:12.041482 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:12.041345    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25" event={"ID":"efd6d54d-0ff7-4da0-b03e-4ffd9efc3505","Type":"ContainerDied","Data":"01b0232f900eb0a690e7b74eb89e8e996336bf962ee830a7f70af74ae5bbcd6d"}
Aug 04 18:57:12.041482 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:12.041404    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25" event={"ID":"efd6d54d-0ff7-4da0-b03e-4ffd9efc3505","Type":"ContainerDied","Data":"921e1462b68b81c2a092ee69ac37968526c16667b3d839c77f5a31e385e5fb02"}
Aug 04 18:57:12.041482 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:12.041438    2576 scope.go:117] "RemoveContainer" containerID="01b0232f900eb0a690e7b74eb89e8e996336bf962ee830a7f70af74ae5bbcd6d"
Aug 04 18:57:12.056756 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:12.056733    2576 scope.go:117] "RemoveContainer" containerID="01b0232f900eb0a690e7b74eb89e8e996336bf962ee830a7f70af74ae5bbcd6d"
Aug 04 18:57:12.057078 ip-10-0-133-187 kubenswrapper[2576]: E0804 18:57:12.057048    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b0232f900eb0a690e7b74eb89e8e996336bf962ee830a7f70af74ae5bbcd6d\": container with ID starting with 01b0232f900eb0a690e7b74eb89e8e996336bf962ee830a7f70af74ae5bbcd6d not found: ID does not exist" containerID="01b0232f900eb0a690e7b74eb89e8e996336bf962ee830a7f70af74ae5bbcd6d"
Aug 04 18:57:12.057156 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:12.057086    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b0232f900eb0a690e7b74eb89e8e996336bf962ee830a7f70af74ae5bbcd6d"} err="failed to get container status \"01b0232f900eb0a690e7b74eb89e8e996336bf962ee830a7f70af74ae5bbcd6d\": rpc error: code = NotFound desc = could not find container \"01b0232f900eb0a690e7b74eb89e8e996336bf962ee830a7f70af74ae5bbcd6d\": container with ID starting with 01b0232f900eb0a690e7b74eb89e8e996336bf962ee830a7f70af74ae5bbcd6d not found: ID does not exist"
Aug 04 18:57:12.069380 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:12.069352    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"]
Aug 04 18:57:12.074518 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:12.074494    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-xpp25"]
Aug 04 18:57:12.581678 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:12.581644    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efd6d54d-0ff7-4da0-b03e-4ffd9efc3505" path="/var/lib/kubelet/pods/efd6d54d-0ff7-4da0-b03e-4ffd9efc3505/volumes"
Aug 04 18:57:16.504286 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:16.504255    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:16.610291 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:16.610251    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:18.895455 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:18.895421    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:19.694929 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.694888    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:19.827280 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.827230    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"]
Aug 04 18:57:19.827677 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.827659    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d64017e-bb81-409a-bb6c-ede0393455ca" containerName="main"
Aug 04 18:57:19.827677 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.827675    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d64017e-bb81-409a-bb6c-ede0393455ca" containerName="main"
Aug 04 18:57:19.827824 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.827693    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efd6d54d-0ff7-4da0-b03e-4ffd9efc3505" containerName="main"
Aug 04 18:57:19.827824 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.827698    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="efd6d54d-0ff7-4da0-b03e-4ffd9efc3505" containerName="main"
Aug 04 18:57:19.827824 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.827765    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d64017e-bb81-409a-bb6c-ede0393455ca" containerName="main"
Aug 04 18:57:19.827824 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.827776    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="efd6d54d-0ff7-4da0-b03e-4ffd9efc3505" containerName="main"
Aug 04 18:57:19.830758 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.830739    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"
Aug 04 18:57:19.835272 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.835251    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Aug 04 18:57:19.835991 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.835971    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Aug 04 18:57:19.837744 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.837725    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-vglqf\""
Aug 04 18:57:19.839076 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.839053    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-qrsn7\""
Aug 04 18:57:19.852251 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.852231    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Aug 04 18:57:19.880550 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.880524    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"]
Aug 04 18:57:19.899320 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.899293    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:19.966411 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.966334    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzzr5\" (UniqueName: \"kubernetes.io/projected/90e31b37-2cb0-43e6-a7ef-050260695409-kube-api-access-gzzr5\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55\" (UID: \"90e31b37-2cb0-43e6-a7ef-050260695409\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"
Aug 04 18:57:19.966411 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:19.966381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90e31b37-2cb0-43e6-a7ef-050260695409-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55\" (UID: \"90e31b37-2cb0-43e6-a7ef-050260695409\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"
Aug 04 18:57:20.067652 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:20.067612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzzr5\" (UniqueName: \"kubernetes.io/projected/90e31b37-2cb0-43e6-a7ef-050260695409-kube-api-access-gzzr5\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55\" (UID: \"90e31b37-2cb0-43e6-a7ef-050260695409\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"
Aug 04 18:57:20.067652 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:20.067655    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90e31b37-2cb0-43e6-a7ef-050260695409-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55\" (UID: \"90e31b37-2cb0-43e6-a7ef-050260695409\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"
Aug 04 18:57:20.070256 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:20.070226    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90e31b37-2cb0-43e6-a7ef-050260695409-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55\" (UID: \"90e31b37-2cb0-43e6-a7ef-050260695409\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"
Aug 04 18:57:20.079249 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:20.079221    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzzr5\" (UniqueName: \"kubernetes.io/projected/90e31b37-2cb0-43e6-a7ef-050260695409-kube-api-access-gzzr5\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55\" (UID: \"90e31b37-2cb0-43e6-a7ef-050260695409\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"
Aug 04 18:57:20.142095 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:20.142052    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"
Aug 04 18:57:20.286497 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:20.286472    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"]
Aug 04 18:57:20.289233 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:57:20.289205    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e31b37_2cb0_43e6_a7ef_050260695409.slice/crio-62e8c20c0f6fd1eb5ae9645ba0c902fd903a114dae05fd381c28bfba47dd8c85 WatchSource:0}: Error finding container 62e8c20c0f6fd1eb5ae9645ba0c902fd903a114dae05fd381c28bfba47dd8c85: Status 404 returned error can't find the container with id 62e8c20c0f6fd1eb5ae9645ba0c902fd903a114dae05fd381c28bfba47dd8c85
Aug 04 18:57:21.086421 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:21.086385    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55" event={"ID":"90e31b37-2cb0-43e6-a7ef-050260695409","Type":"ContainerStarted","Data":"215400bfb1399f5ca188651afe6a2960cd926c36f0a74279621ed5459122a4b1"}
Aug 04 18:57:21.086900 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:21.086448    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55" event={"ID":"90e31b37-2cb0-43e6-a7ef-050260695409","Type":"ContainerStarted","Data":"62e8c20c0f6fd1eb5ae9645ba0c902fd903a114dae05fd381c28bfba47dd8c85"}
Aug 04 18:57:21.086900 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:21.086528    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"
Aug 04 18:57:23.560836 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:23.560804    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:24.541957 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:24.541923    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:24.642327 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:24.642289    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:25.050189 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.050152    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:25.296772 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.296718    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55" podStartSLOduration=6.296699392 podStartE2EDuration="6.296699392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:57:21.114367749 +0000 UTC m=+3441.090199447" watchObservedRunningTime="2026-08-04 18:57:25.296699392 +0000 UTC m=+3445.272531089"
Aug 04 18:57:25.298071 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.298050    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"]
Aug 04 18:57:25.301717 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.301672    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"
Aug 04 18:57:25.305885 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.305857    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Aug 04 18:57:25.305885 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.305879    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Aug 04 18:57:25.306043 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.305927    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-njgj7\""
Aug 04 18:57:25.306043 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.305924    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Aug 04 18:57:25.306043 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.305871    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-nnrcd\""
Aug 04 18:57:25.309977 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.309955    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbr9s\" (UniqueName: \"kubernetes.io/projected/54f4ebc0-aeba-45d9-98b7-676ef50ef6d5-kube-api-access-hbr9s\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt\" (UID: \"54f4ebc0-aeba-45d9-98b7-676ef50ef6d5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"
Aug 04 18:57:25.310073 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.309999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f4ebc0-aeba-45d9-98b7-676ef50ef6d5-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt\" (UID: \"54f4ebc0-aeba-45d9-98b7-676ef50ef6d5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"
Aug 04 18:57:25.318276 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.318254    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"]
Aug 04 18:57:25.341394 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.341358    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:25.411158 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.411105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbr9s\" (UniqueName: \"kubernetes.io/projected/54f4ebc0-aeba-45d9-98b7-676ef50ef6d5-kube-api-access-hbr9s\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt\" (UID: \"54f4ebc0-aeba-45d9-98b7-676ef50ef6d5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"
Aug 04 18:57:25.411328 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.411177    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f4ebc0-aeba-45d9-98b7-676ef50ef6d5-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt\" (UID: \"54f4ebc0-aeba-45d9-98b7-676ef50ef6d5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"
Aug 04 18:57:25.413597 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.413577    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f4ebc0-aeba-45d9-98b7-676ef50ef6d5-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt\" (UID: \"54f4ebc0-aeba-45d9-98b7-676ef50ef6d5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"
Aug 04 18:57:25.421882 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.421853    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbr9s\" (UniqueName: \"kubernetes.io/projected/54f4ebc0-aeba-45d9-98b7-676ef50ef6d5-kube-api-access-hbr9s\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt\" (UID: \"54f4ebc0-aeba-45d9-98b7-676ef50ef6d5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"
Aug 04 18:57:25.612842 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.612754    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"
Aug 04 18:57:25.760936 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:25.760907    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"]
Aug 04 18:57:25.764078 ip-10-0-133-187 kubenswrapper[2576]: W0804 18:57:25.764042    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f4ebc0_aeba_45d9_98b7_676ef50ef6d5.slice/crio-97c8f7d9ccf4f9ce5b61dd68a3c9299bf72016b0eb4cce25315d7c303ea9b5ac WatchSource:0}: Error finding container 97c8f7d9ccf4f9ce5b61dd68a3c9299bf72016b0eb4cce25315d7c303ea9b5ac: Status 404 returned error can't find the container with id 97c8f7d9ccf4f9ce5b61dd68a3c9299bf72016b0eb4cce25315d7c303ea9b5ac
Aug 04 18:57:26.112026 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:26.111985    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt" event={"ID":"54f4ebc0-aeba-45d9-98b7-676ef50ef6d5","Type":"ContainerStarted","Data":"881269af5f38703fdb097a86ef964efa2281baa48b6c47792ffd6724e819f80a"}
Aug 04 18:57:26.112268 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:26.112037    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt" event={"ID":"54f4ebc0-aeba-45d9-98b7-676ef50ef6d5","Type":"ContainerStarted","Data":"97c8f7d9ccf4f9ce5b61dd68a3c9299bf72016b0eb4cce25315d7c303ea9b5ac"}
Aug 04 18:57:26.112268 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:26.112066    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"
Aug 04 18:57:26.133656 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:26.133591    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt" podStartSLOduration=1.133571714 podStartE2EDuration="1.133571714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:57:26.132077777 +0000 UTC m=+3446.107909475" watchObservedRunningTime="2026-08-04 18:57:26.133571714 +0000 UTC m=+3446.109403411"
Aug 04 18:57:29.412198 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:29.412160    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:29.542031 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:29.541995    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:32.796523 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:32.796491    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:40.438854 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:40.438819    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:57:52.093413 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:52.093378    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"
Aug 04 18:57:57.117909 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:57:57.117876    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"
Aug 04 18:58:41.544041 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:58:41.544006    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:58:43.805181 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:58:43.805149    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 18:58:48.550684 ip-10-0-133-187 kubenswrapper[2576]: I0804 18:58:48.550617    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:06.191733 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:06.191700    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:14.078154 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:14.078096    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:14.128259 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:14.128224    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"]
Aug 04 19:00:14.201138 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:14.201091    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:16.493051 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:16.493003    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:22.093262 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:22.093222    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55" podUID="90e31b37-2cb0-43e6-a7ef-050260695409" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 19:00:26.388741 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:26.388704    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"]
Aug 04 19:00:26.392773 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:26.392734    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:26.507561 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:26.507520    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:27.117776 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:27.117735    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt" podUID="54f4ebc0-aeba-45d9-98b7-676ef50ef6d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 19:00:29.147350 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.147303    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55" podUID="90e31b37-2cb0-43e6-a7ef-050260695409" containerName="main" containerID="cri-o://215400bfb1399f5ca188651afe6a2960cd926c36f0a74279621ed5459122a4b1" gracePeriod=45
Aug 04 19:00:29.390888 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.390857    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"
Aug 04 19:00:29.470963 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.470879    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzzr5\" (UniqueName: \"kubernetes.io/projected/90e31b37-2cb0-43e6-a7ef-050260695409-kube-api-access-gzzr5\") pod \"90e31b37-2cb0-43e6-a7ef-050260695409\" (UID: \"90e31b37-2cb0-43e6-a7ef-050260695409\") "
Aug 04 19:00:29.470963 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.470913    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90e31b37-2cb0-43e6-a7ef-050260695409-tls-certs\") pod \"90e31b37-2cb0-43e6-a7ef-050260695409\" (UID: \"90e31b37-2cb0-43e6-a7ef-050260695409\") "
Aug 04 19:00:29.473210 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.473181    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e31b37-2cb0-43e6-a7ef-050260695409-kube-api-access-gzzr5" (OuterVolumeSpecName: "kube-api-access-gzzr5") pod "90e31b37-2cb0-43e6-a7ef-050260695409" (UID: "90e31b37-2cb0-43e6-a7ef-050260695409"). InnerVolumeSpecName "kube-api-access-gzzr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:00:29.473210 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.473182    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90e31b37-2cb0-43e6-a7ef-050260695409-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90e31b37-2cb0-43e6-a7ef-050260695409" (UID: "90e31b37-2cb0-43e6-a7ef-050260695409"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 19:00:29.572105 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.572072    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzzr5\" (UniqueName: \"kubernetes.io/projected/90e31b37-2cb0-43e6-a7ef-050260695409-kube-api-access-gzzr5\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 19:00:29.572285 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.572103    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90e31b37-2cb0-43e6-a7ef-050260695409-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 19:00:29.978423 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.978389    2576 generic.go:356] "Generic (PLEG): container finished" podID="90e31b37-2cb0-43e6-a7ef-050260695409" containerID="215400bfb1399f5ca188651afe6a2960cd926c36f0a74279621ed5459122a4b1" exitCode=0
Aug 04 19:00:29.978631 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.978429    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"
Aug 04 19:00:29.978631 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.978469    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55" event={"ID":"90e31b37-2cb0-43e6-a7ef-050260695409","Type":"ContainerDied","Data":"215400bfb1399f5ca188651afe6a2960cd926c36f0a74279621ed5459122a4b1"}
Aug 04 19:00:29.978631 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.978524    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55" event={"ID":"90e31b37-2cb0-43e6-a7ef-050260695409","Type":"ContainerDied","Data":"62e8c20c0f6fd1eb5ae9645ba0c902fd903a114dae05fd381c28bfba47dd8c85"}
Aug 04 19:00:29.978631 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.978553    2576 scope.go:117] "RemoveContainer" containerID="215400bfb1399f5ca188651afe6a2960cd926c36f0a74279621ed5459122a4b1"
Aug 04 19:00:29.987491 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.987474    2576 scope.go:117] "RemoveContainer" containerID="215400bfb1399f5ca188651afe6a2960cd926c36f0a74279621ed5459122a4b1"
Aug 04 19:00:29.987766 ip-10-0-133-187 kubenswrapper[2576]: E0804 19:00:29.987747    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215400bfb1399f5ca188651afe6a2960cd926c36f0a74279621ed5459122a4b1\": container with ID starting with 215400bfb1399f5ca188651afe6a2960cd926c36f0a74279621ed5459122a4b1 not found: ID does not exist" containerID="215400bfb1399f5ca188651afe6a2960cd926c36f0a74279621ed5459122a4b1"
Aug 04 19:00:29.987815 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.987774    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215400bfb1399f5ca188651afe6a2960cd926c36f0a74279621ed5459122a4b1"} err="failed to get container status \"215400bfb1399f5ca188651afe6a2960cd926c36f0a74279621ed5459122a4b1\": rpc error: code = NotFound desc = could not find container \"215400bfb1399f5ca188651afe6a2960cd926c36f0a74279621ed5459122a4b1\": container with ID starting with 215400bfb1399f5ca188651afe6a2960cd926c36f0a74279621ed5459122a4b1 not found: ID does not exist"
Aug 04 19:00:29.999387 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:29.999359    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"]
Aug 04 19:00:30.003384 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:30.003356    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d74f8g55"]
Aug 04 19:00:30.582128 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:30.582075    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90e31b37-2cb0-43e6-a7ef-050260695409" path="/var/lib/kubelet/pods/90e31b37-2cb0-43e6-a7ef-050260695409/volumes"
Aug 04 19:00:34.330461 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:34.330421    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:34.437160 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:34.437100    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:34.549256 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:34.549202    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:34.650072 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:34.649984    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:37.117761 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:37.117718    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt" podUID="54f4ebc0-aeba-45d9-98b7-676ef50ef6d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 19:00:38.522718 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:38.522680    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:40.710680 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:40.710648    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:41.408457 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:41.408390    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt" podUID="54f4ebc0-aeba-45d9-98b7-676ef50ef6d5" containerName="main" containerID="cri-o://881269af5f38703fdb097a86ef964efa2281baa48b6c47792ffd6724e819f80a" gracePeriod=45
Aug 04 19:00:41.655817 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:41.655792    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"
Aug 04 19:00:41.784506 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:41.784412    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f4ebc0-aeba-45d9-98b7-676ef50ef6d5-tls-certs\") pod \"54f4ebc0-aeba-45d9-98b7-676ef50ef6d5\" (UID: \"54f4ebc0-aeba-45d9-98b7-676ef50ef6d5\") "
Aug 04 19:00:41.784949 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:41.784526    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbr9s\" (UniqueName: \"kubernetes.io/projected/54f4ebc0-aeba-45d9-98b7-676ef50ef6d5-kube-api-access-hbr9s\") pod \"54f4ebc0-aeba-45d9-98b7-676ef50ef6d5\" (UID: \"54f4ebc0-aeba-45d9-98b7-676ef50ef6d5\") "
Aug 04 19:00:41.786622 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:41.786591    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f4ebc0-aeba-45d9-98b7-676ef50ef6d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54f4ebc0-aeba-45d9-98b7-676ef50ef6d5" (UID: "54f4ebc0-aeba-45d9-98b7-676ef50ef6d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 19:00:41.786741 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:41.786647    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f4ebc0-aeba-45d9-98b7-676ef50ef6d5-kube-api-access-hbr9s" (OuterVolumeSpecName: "kube-api-access-hbr9s") pod "54f4ebc0-aeba-45d9-98b7-676ef50ef6d5" (UID: "54f4ebc0-aeba-45d9-98b7-676ef50ef6d5"). InnerVolumeSpecName "kube-api-access-hbr9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:00:41.886104 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:41.886068    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbr9s\" (UniqueName: \"kubernetes.io/projected/54f4ebc0-aeba-45d9-98b7-676ef50ef6d5-kube-api-access-hbr9s\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 19:00:41.886104 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:41.886099    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f4ebc0-aeba-45d9-98b7-676ef50ef6d5-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 19:00:42.036392 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:42.036302    2576 generic.go:356] "Generic (PLEG): container finished" podID="54f4ebc0-aeba-45d9-98b7-676ef50ef6d5" containerID="881269af5f38703fdb097a86ef964efa2281baa48b6c47792ffd6724e819f80a" exitCode=0
Aug 04 19:00:42.036392 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:42.036363    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"
Aug 04 19:00:42.036579 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:42.036389    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt" event={"ID":"54f4ebc0-aeba-45d9-98b7-676ef50ef6d5","Type":"ContainerDied","Data":"881269af5f38703fdb097a86ef964efa2281baa48b6c47792ffd6724e819f80a"}
Aug 04 19:00:42.036579 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:42.036470    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt" event={"ID":"54f4ebc0-aeba-45d9-98b7-676ef50ef6d5","Type":"ContainerDied","Data":"97c8f7d9ccf4f9ce5b61dd68a3c9299bf72016b0eb4cce25315d7c303ea9b5ac"}
Aug 04 19:00:42.036579 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:42.036493    2576 scope.go:117] "RemoveContainer" containerID="881269af5f38703fdb097a86ef964efa2281baa48b6c47792ffd6724e819f80a"
Aug 04 19:00:42.045773 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:42.045749    2576 scope.go:117] "RemoveContainer" containerID="881269af5f38703fdb097a86ef964efa2281baa48b6c47792ffd6724e819f80a"
Aug 04 19:00:42.046219 ip-10-0-133-187 kubenswrapper[2576]: E0804 19:00:42.046049    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"881269af5f38703fdb097a86ef964efa2281baa48b6c47792ffd6724e819f80a\": container with ID starting with 881269af5f38703fdb097a86ef964efa2281baa48b6c47792ffd6724e819f80a not found: ID does not exist" containerID="881269af5f38703fdb097a86ef964efa2281baa48b6c47792ffd6724e819f80a"
Aug 04 19:00:42.046219 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:42.046088    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"881269af5f38703fdb097a86ef964efa2281baa48b6c47792ffd6724e819f80a"} err="failed to get container status \"881269af5f38703fdb097a86ef964efa2281baa48b6c47792ffd6724e819f80a\": rpc error: code = NotFound desc = could not find container \"881269af5f38703fdb097a86ef964efa2281baa48b6c47792ffd6724e819f80a\": container with ID starting with 881269af5f38703fdb097a86ef964efa2281baa48b6c47792ffd6724e819f80a not found: ID does not exist"
Aug 04 19:00:42.059974 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:42.059943    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"]
Aug 04 19:00:42.063615 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:42.063593    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-wsbqt"]
Aug 04 19:00:42.582494 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:42.582460    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54f4ebc0-aeba-45d9-98b7-676ef50ef6d5" path="/var/lib/kubelet/pods/54f4ebc0-aeba-45d9-98b7-676ef50ef6d5/volumes"
Aug 04 19:00:46.421637 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.421602    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:46.646987 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.646955    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:46.750003 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.749914    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:46.820127 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.820085    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"]
Aug 04 19:00:46.820549 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.820529    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54f4ebc0-aeba-45d9-98b7-676ef50ef6d5" containerName="main"
Aug 04 19:00:46.820649 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.820552    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="54f4ebc0-aeba-45d9-98b7-676ef50ef6d5" containerName="main"
Aug 04 19:00:46.820649 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.820581    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90e31b37-2cb0-43e6-a7ef-050260695409" containerName="main"
Aug 04 19:00:46.820649 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.820589    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90e31b37-2cb0-43e6-a7ef-050260695409" containerName="main"
Aug 04 19:00:46.820802 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.820690    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="54f4ebc0-aeba-45d9-98b7-676ef50ef6d5" containerName="main"
Aug 04 19:00:46.820802 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.820705    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="90e31b37-2cb0-43e6-a7ef-050260695409" containerName="main"
Aug 04 19:00:46.823815 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.823793    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"
Aug 04 19:00:46.828347 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.828323    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Aug 04 19:00:46.828495 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.828379    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-m65hb\""
Aug 04 19:00:46.828495 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.828395    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-k7wgs\""
Aug 04 19:00:46.828495 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.828399    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Aug 04 19:00:46.828495 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.828457    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Aug 04 19:00:46.845026 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.844997    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"]
Aug 04 19:00:46.936522 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.936454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcqgh\" (UniqueName: \"kubernetes.io/projected/928bd27a-d62d-4318-98a2-922d902f70ba-kube-api-access-tcqgh\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc\" (UID: \"928bd27a-d62d-4318-98a2-922d902f70ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"
Aug 04 19:00:46.936714 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:46.936549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928bd27a-d62d-4318-98a2-922d902f70ba-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc\" (UID: \"928bd27a-d62d-4318-98a2-922d902f70ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"
Aug 04 19:00:47.037273 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:47.037243    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcqgh\" (UniqueName: \"kubernetes.io/projected/928bd27a-d62d-4318-98a2-922d902f70ba-kube-api-access-tcqgh\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc\" (UID: \"928bd27a-d62d-4318-98a2-922d902f70ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"
Aug 04 19:00:47.037437 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:47.037299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928bd27a-d62d-4318-98a2-922d902f70ba-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc\" (UID: \"928bd27a-d62d-4318-98a2-922d902f70ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"
Aug 04 19:00:47.039982 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:47.039946    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928bd27a-d62d-4318-98a2-922d902f70ba-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc\" (UID: \"928bd27a-d62d-4318-98a2-922d902f70ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"
Aug 04 19:00:47.052522 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:47.052489    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcqgh\" (UniqueName: \"kubernetes.io/projected/928bd27a-d62d-4318-98a2-922d902f70ba-kube-api-access-tcqgh\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc\" (UID: \"928bd27a-d62d-4318-98a2-922d902f70ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"
Aug 04 19:00:47.134721 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:47.134682    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"
Aug 04 19:00:47.223235 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:47.223200    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:47.274185 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:47.274156    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"]
Aug 04 19:00:47.275083 ip-10-0-133-187 kubenswrapper[2576]: W0804 19:00:47.275060    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928bd27a_d62d_4318_98a2_922d902f70ba.slice/crio-5c891cbb53eb39f82d8b91b75b1fc58667337c62986d5f602e404cd61c774df5 WatchSource:0}: Error finding container 5c891cbb53eb39f82d8b91b75b1fc58667337c62986d5f602e404cd61c774df5: Status 404 returned error can't find the container with id 5c891cbb53eb39f82d8b91b75b1fc58667337c62986d5f602e404cd61c774df5
Aug 04 19:00:47.324373 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:47.324348    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:00:48.066784 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:48.066746    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc" event={"ID":"928bd27a-d62d-4318-98a2-922d902f70ba","Type":"ContainerStarted","Data":"af404228a1d48eb0ab532d59f301685a2833d8a48af03e613016ad1730ff9d16"}
Aug 04 19:00:48.067311 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:48.066806    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc" event={"ID":"928bd27a-d62d-4318-98a2-922d902f70ba","Type":"ContainerStarted","Data":"5c891cbb53eb39f82d8b91b75b1fc58667337c62986d5f602e404cd61c774df5"}
Aug 04 19:00:48.067311 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:48.066854    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"
Aug 04 19:00:48.083186 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:00:48.083131    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc" podStartSLOduration=2.083098335 podStartE2EDuration="2.083098335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:00:48.082176997 +0000 UTC m=+3648.058008698" watchObservedRunningTime="2026-08-04 19:00:48.083098335 +0000 UTC m=+3648.058930031"
Aug 04 19:01:19.073471 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:19.073435    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"
Aug 04 19:01:31.560517 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:31.560437    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:01:31.648313 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:31.648275    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:01:37.599418 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:37.599369    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:01:37.760540 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:37.760499    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"]
Aug 04 19:01:37.764381 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:37.764355    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"
Aug 04 19:01:37.768286 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:37.768249    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Aug 04 19:01:37.768286 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:37.768267    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-8z9pv\""
Aug 04 19:01:37.768513 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:37.768250    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-lpzq8\""
Aug 04 19:01:37.768513 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:37.768357    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Aug 04 19:01:37.768606 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:37.768556    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Aug 04 19:01:37.776194 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:37.776168    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"]
Aug 04 19:01:37.796815 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:37.796786    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:01:37.908079 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:37.907989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2sdc\" (UniqueName: \"kubernetes.io/projected/f2dde748-3196-4303-b1f3-5b64c4b2b59a-kube-api-access-p2sdc\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864\" (UID: \"f2dde748-3196-4303-b1f3-5b64c4b2b59a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"
Aug 04 19:01:37.908079 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:37.908033    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2dde748-3196-4303-b1f3-5b64c4b2b59a-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864\" (UID: \"f2dde748-3196-4303-b1f3-5b64c4b2b59a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"
Aug 04 19:01:38.009488 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:38.009449    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2sdc\" (UniqueName: \"kubernetes.io/projected/f2dde748-3196-4303-b1f3-5b64c4b2b59a-kube-api-access-p2sdc\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864\" (UID: \"f2dde748-3196-4303-b1f3-5b64c4b2b59a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"
Aug 04 19:01:38.009488 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:38.009493    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2dde748-3196-4303-b1f3-5b64c4b2b59a-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864\" (UID: \"f2dde748-3196-4303-b1f3-5b64c4b2b59a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"
Aug 04 19:01:38.012034 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:38.012008    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2dde748-3196-4303-b1f3-5b64c4b2b59a-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864\" (UID: \"f2dde748-3196-4303-b1f3-5b64c4b2b59a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"
Aug 04 19:01:38.017660 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:38.017632    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2sdc\" (UniqueName: \"kubernetes.io/projected/f2dde748-3196-4303-b1f3-5b64c4b2b59a-kube-api-access-p2sdc\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864\" (UID: \"f2dde748-3196-4303-b1f3-5b64c4b2b59a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"
Aug 04 19:01:38.078164 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:38.078107    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"
Aug 04 19:01:38.202836 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:38.202807    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:01:38.211629 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:38.211603    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"]
Aug 04 19:01:38.214258 ip-10-0-133-187 kubenswrapper[2576]: W0804 19:01:38.214224    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2dde748_3196_4303_b1f3_5b64c4b2b59a.slice/crio-6a9017d627c843ba701b7b92a8000fccbbb3940485759af17696476b70ab1ea0 WatchSource:0}: Error finding container 6a9017d627c843ba701b7b92a8000fccbbb3940485759af17696476b70ab1ea0: Status 404 returned error can't find the container with id 6a9017d627c843ba701b7b92a8000fccbbb3940485759af17696476b70ab1ea0
Aug 04 19:01:38.297578 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:38.297539    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864" event={"ID":"f2dde748-3196-4303-b1f3-5b64c4b2b59a","Type":"ContainerStarted","Data":"094887cc6726f20251b2fbb45963cdd0f774ff0d62a65d3e726520a2c2a53c17"}
Aug 04 19:01:38.297739 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:38.297607    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864" event={"ID":"f2dde748-3196-4303-b1f3-5b64c4b2b59a","Type":"ContainerStarted","Data":"6a9017d627c843ba701b7b92a8000fccbbb3940485759af17696476b70ab1ea0"}
Aug 04 19:01:38.297739 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:38.297673    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"
Aug 04 19:01:38.315609 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:38.315555    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864" podStartSLOduration=1.315538591 podStartE2EDuration="1.315538591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:01:38.315047466 +0000 UTC m=+3698.290879164" watchObservedRunningTime="2026-08-04 19:01:38.315538591 +0000 UTC m=+3698.291370287"
Aug 04 19:01:44.280091 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:44.280056    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:01:44.354609 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:44.354572    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:01:53.529029 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:01:53.528978    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:02:09.303604 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:09.303571    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"
Aug 04 19:02:10.735122 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.735077    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"]
Aug 04 19:02:10.738832 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.738808    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.754254 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.754218    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"]
Aug 04 19:02:10.810667 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.810633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrgg5\" (UniqueName: \"kubernetes.io/projected/ba956419-88c0-4184-bfc4-f44283b3745d-kube-api-access-vrgg5\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.810851 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.810679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba956419-88c0-4184-bfc4-f44283b3745d-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.810851 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.810756    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.810851 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.810784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.810851 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.810808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.810851 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.810833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.912040 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.911992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.912246 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.912059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrgg5\" (UniqueName: \"kubernetes.io/projected/ba956419-88c0-4184-bfc4-f44283b3745d-kube-api-access-vrgg5\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.912246 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.912100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba956419-88c0-4184-bfc4-f44283b3745d-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.912246 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.912208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.912396 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.912246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.912396 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.912293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.912604 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.912579    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.912691 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.912669    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.912746 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.912691    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.914407 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.914388    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.914706 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.914686    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba956419-88c0-4184-bfc4-f44283b3745d-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:10.922660 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:10.922633    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrgg5\" (UniqueName: \"kubernetes.io/projected/ba956419-88c0-4184-bfc4-f44283b3745d-kube-api-access-vrgg5\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-h5x25\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:11.050074 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:11.050043    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:11.189746 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:11.189719    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"]
Aug 04 19:02:11.192624 ip-10-0-133-187 kubenswrapper[2576]: W0804 19:02:11.192596    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba956419_88c0_4184_bfc4_f44283b3745d.slice/crio-7da77f0e95248b1d8e8cb72f362e74c9d138d2fb7831c3099bb307d2a2e7e3b0 WatchSource:0}: Error finding container 7da77f0e95248b1d8e8cb72f362e74c9d138d2fb7831c3099bb307d2a2e7e3b0: Status 404 returned error can't find the container with id 7da77f0e95248b1d8e8cb72f362e74c9d138d2fb7831c3099bb307d2a2e7e3b0
Aug 04 19:02:11.456079 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:11.455987    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25" event={"ID":"ba956419-88c0-4184-bfc4-f44283b3745d","Type":"ContainerStarted","Data":"fbff6bebc7118c968822ea196d15e898847d0a7a91321aa0b814c52e710c77c4"}
Aug 04 19:02:11.456079 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:11.456052    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25" event={"ID":"ba956419-88c0-4184-bfc4-f44283b3745d","Type":"ContainerStarted","Data":"7da77f0e95248b1d8e8cb72f362e74c9d138d2fb7831c3099bb307d2a2e7e3b0"}
Aug 04 19:02:11.479795 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:11.479741    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25" podStartSLOduration=1.479726964 podStartE2EDuration="1.479726964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:02:11.477087284 +0000 UTC m=+3731.452919007" watchObservedRunningTime="2026-08-04 19:02:11.479726964 +0000 UTC m=+3731.455558661"
Aug 04 19:02:12.051197 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:12.051164    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:17.325704 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.325669    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:02:17.334571 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.334533    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"]
Aug 04 19:02:17.336345 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.336319    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"]
Aug 04 19:02:17.344742 ip-10-0-133-187 kubenswrapper[2576]: E0804 19:02:17.344707    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 19:02:17.344742 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 19:02:17.344742 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 19:02:17.344742 ip-10-0-133-187 kubenswrapper[2576]:  > containerID="fbff6bebc7118c968822ea196d15e898847d0a7a91321aa0b814c52e710c77c4" cmd=["/bin/sleep","15"]
Aug 04 19:02:17.344983 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.344762    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25" message="" err=<
Aug 04 19:02:17.344983 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 19:02:17.344983 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 19:02:17.344983 ip-10-0-133-187 kubenswrapper[2576]:  >
Aug 04 19:02:17.344983 ip-10-0-133-187 kubenswrapper[2576]: E0804 19:02:17.344780    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 19:02:17.344983 ip-10-0-133-187 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 19:02:17.344983 ip-10-0-133-187 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Aug 04 19:02:17.344983 ip-10-0-133-187 kubenswrapper[2576]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25" podUID="ba956419-88c0-4184-bfc4-f44283b3745d" containerName="main" containerID="cri-o://fbff6bebc7118c968822ea196d15e898847d0a7a91321aa0b814c52e710c77c4"
Aug 04 19:02:17.344983 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.344828    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25" podUID="ba956419-88c0-4184-bfc4-f44283b3745d" containerName="main" containerID="cri-o://fbff6bebc7118c968822ea196d15e898847d0a7a91321aa0b814c52e710c77c4" gracePeriod=60
Aug 04 19:02:17.435509 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.435462    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:02:17.489615 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.489579    2576 generic.go:356] "Generic (PLEG): container finished" podID="ba956419-88c0-4184-bfc4-f44283b3745d" containerID="fbff6bebc7118c968822ea196d15e898847d0a7a91321aa0b814c52e710c77c4" exitCode=0
Aug 04 19:02:17.489790 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.489647    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25" event={"ID":"ba956419-88c0-4184-bfc4-f44283b3745d","Type":"ContainerDied","Data":"fbff6bebc7118c968822ea196d15e898847d0a7a91321aa0b814c52e710c77c4"}
Aug 04 19:02:17.606286 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.606260    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:17.674743 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.674701    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-home\") pod \"ba956419-88c0-4184-bfc4-f44283b3745d\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") "
Aug 04 19:02:17.674743 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.674749    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-model-cache\") pod \"ba956419-88c0-4184-bfc4-f44283b3745d\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") "
Aug 04 19:02:17.675014 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.674774    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-tmp-dir\") pod \"ba956419-88c0-4184-bfc4-f44283b3745d\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") "
Aug 04 19:02:17.675014 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.674802    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrgg5\" (UniqueName: \"kubernetes.io/projected/ba956419-88c0-4184-bfc4-f44283b3745d-kube-api-access-vrgg5\") pod \"ba956419-88c0-4184-bfc4-f44283b3745d\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") "
Aug 04 19:02:17.675014 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.674841    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba956419-88c0-4184-bfc4-f44283b3745d-tls-certs\") pod \"ba956419-88c0-4184-bfc4-f44283b3745d\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") "
Aug 04 19:02:17.675014 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.674934    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-dshm\") pod \"ba956419-88c0-4184-bfc4-f44283b3745d\" (UID: \"ba956419-88c0-4184-bfc4-f44283b3745d\") "
Aug 04 19:02:17.675263 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.675020    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-home" (OuterVolumeSpecName: "home") pod "ba956419-88c0-4184-bfc4-f44283b3745d" (UID: "ba956419-88c0-4184-bfc4-f44283b3745d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:02:17.675263 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.675079    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba956419-88c0-4184-bfc4-f44283b3745d" (UID: "ba956419-88c0-4184-bfc4-f44283b3745d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:02:17.675263 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.675088    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba956419-88c0-4184-bfc4-f44283b3745d" (UID: "ba956419-88c0-4184-bfc4-f44283b3745d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:02:17.675417 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.675317    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 19:02:17.675417 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.675338    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 19:02:17.675417 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.675353    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 19:02:17.677343 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.677312    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba956419-88c0-4184-bfc4-f44283b3745d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba956419-88c0-4184-bfc4-f44283b3745d" (UID: "ba956419-88c0-4184-bfc4-f44283b3745d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 19:02:17.677587 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.677564    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-dshm" (OuterVolumeSpecName: "dshm") pod "ba956419-88c0-4184-bfc4-f44283b3745d" (UID: "ba956419-88c0-4184-bfc4-f44283b3745d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:02:17.677671 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.677581    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba956419-88c0-4184-bfc4-f44283b3745d-kube-api-access-vrgg5" (OuterVolumeSpecName: "kube-api-access-vrgg5") pod "ba956419-88c0-4184-bfc4-f44283b3745d" (UID: "ba956419-88c0-4184-bfc4-f44283b3745d"). InnerVolumeSpecName "kube-api-access-vrgg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:02:17.775829 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.775793    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrgg5\" (UniqueName: \"kubernetes.io/projected/ba956419-88c0-4184-bfc4-f44283b3745d-kube-api-access-vrgg5\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 19:02:17.775829 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.775824    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba956419-88c0-4184-bfc4-f44283b3745d-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 19:02:17.775829 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:17.775833    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba956419-88c0-4184-bfc4-f44283b3745d-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 19:02:18.495330 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:18.495297    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"
Aug 04 19:02:18.495330 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:18.495310    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25" event={"ID":"ba956419-88c0-4184-bfc4-f44283b3745d","Type":"ContainerDied","Data":"7da77f0e95248b1d8e8cb72f362e74c9d138d2fb7831c3099bb307d2a2e7e3b0"}
Aug 04 19:02:18.495920 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:18.495366    2576 scope.go:117] "RemoveContainer" containerID="fbff6bebc7118c968822ea196d15e898847d0a7a91321aa0b814c52e710c77c4"
Aug 04 19:02:18.518062 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:18.518034    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"]
Aug 04 19:02:18.523496 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:18.523464    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-h5x25"]
Aug 04 19:02:18.582262 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:18.582233    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba956419-88c0-4184-bfc4-f44283b3745d" path="/var/lib/kubelet/pods/ba956419-88c0-4184-bfc4-f44283b3745d/volumes"
Aug 04 19:02:19.304498 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:19.304456    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864" podUID="f2dde748-3196-4303-b1f3-5b64c4b2b59a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 19:02:29.303768 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:29.303727    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864" podUID="f2dde748-3196-4303-b1f3-5b64c4b2b59a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 19:02:32.358949 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:32.358884    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864" podUID="f2dde748-3196-4303-b1f3-5b64c4b2b59a" containerName="main" containerID="cri-o://094887cc6726f20251b2fbb45963cdd0f774ff0d62a65d3e726520a2c2a53c17" gracePeriod=45
Aug 04 19:02:32.569072 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:32.569039    2576 generic.go:356] "Generic (PLEG): container finished" podID="f2dde748-3196-4303-b1f3-5b64c4b2b59a" containerID="094887cc6726f20251b2fbb45963cdd0f774ff0d62a65d3e726520a2c2a53c17" exitCode=0
Aug 04 19:02:32.569260 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:32.569134    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864" event={"ID":"f2dde748-3196-4303-b1f3-5b64c4b2b59a","Type":"ContainerDied","Data":"094887cc6726f20251b2fbb45963cdd0f774ff0d62a65d3e726520a2c2a53c17"}
Aug 04 19:02:32.610017 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:32.609951    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"
Aug 04 19:02:32.614630 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:32.614611    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2sdc\" (UniqueName: \"kubernetes.io/projected/f2dde748-3196-4303-b1f3-5b64c4b2b59a-kube-api-access-p2sdc\") pod \"f2dde748-3196-4303-b1f3-5b64c4b2b59a\" (UID: \"f2dde748-3196-4303-b1f3-5b64c4b2b59a\") "
Aug 04 19:02:32.614737 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:32.614726    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2dde748-3196-4303-b1f3-5b64c4b2b59a-tls-certs\") pod \"f2dde748-3196-4303-b1f3-5b64c4b2b59a\" (UID: \"f2dde748-3196-4303-b1f3-5b64c4b2b59a\") "
Aug 04 19:02:32.616705 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:32.616672    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dde748-3196-4303-b1f3-5b64c4b2b59a-kube-api-access-p2sdc" (OuterVolumeSpecName: "kube-api-access-p2sdc") pod "f2dde748-3196-4303-b1f3-5b64c4b2b59a" (UID: "f2dde748-3196-4303-b1f3-5b64c4b2b59a"). InnerVolumeSpecName "kube-api-access-p2sdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:02:32.616812 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:32.616715    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2dde748-3196-4303-b1f3-5b64c4b2b59a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2dde748-3196-4303-b1f3-5b64c4b2b59a" (UID: "f2dde748-3196-4303-b1f3-5b64c4b2b59a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 19:02:32.715800 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:32.715764    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2dde748-3196-4303-b1f3-5b64c4b2b59a-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 19:02:32.715800 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:32.715795    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2sdc\" (UniqueName: \"kubernetes.io/projected/f2dde748-3196-4303-b1f3-5b64c4b2b59a-kube-api-access-p2sdc\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 19:02:33.575232 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:33.575196    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864" event={"ID":"f2dde748-3196-4303-b1f3-5b64c4b2b59a","Type":"ContainerDied","Data":"6a9017d627c843ba701b7b92a8000fccbbb3940485759af17696476b70ab1ea0"}
Aug 04 19:02:33.575232 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:33.575223    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"
Aug 04 19:02:33.575762 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:33.575259    2576 scope.go:117] "RemoveContainer" containerID="094887cc6726f20251b2fbb45963cdd0f774ff0d62a65d3e726520a2c2a53c17"
Aug 04 19:02:33.606341 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:33.606308    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"]
Aug 04 19:02:33.611285 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:33.611260    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-7p864"]
Aug 04 19:02:34.581929 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:34.581892    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2dde748-3196-4303-b1f3-5b64c4b2b59a" path="/var/lib/kubelet/pods/f2dde748-3196-4303-b1f3-5b64c4b2b59a/volumes"
Aug 04 19:02:38.247602 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:38.247567    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:02:44.346663 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:44.346624    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:02:49.966916 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:49.966877    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:02:50.068518 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:02:50.068481    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:03:35.297684 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:35.297646    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"]
Aug 04 19:03:35.303177 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:35.303087    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:03:35.397224 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:35.397187    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:03:39.073770 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:39.073727    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc" podUID="928bd27a-d62d-4318-98a2-922d902f70ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 19:03:49.073412 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:49.073366    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc" podUID="928bd27a-d62d-4318-98a2-922d902f70ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 19:03:50.317810 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.317746    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc" podUID="928bd27a-d62d-4318-98a2-922d902f70ba" containerName="main" containerID="cri-o://af404228a1d48eb0ab532d59f301685a2833d8a48af03e613016ad1730ff9d16" gracePeriod=45
Aug 04 19:03:50.568959 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.568898    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"
Aug 04 19:03:50.737004 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.736968    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928bd27a-d62d-4318-98a2-922d902f70ba-tls-certs\") pod \"928bd27a-d62d-4318-98a2-922d902f70ba\" (UID: \"928bd27a-d62d-4318-98a2-922d902f70ba\") "
Aug 04 19:03:50.737211 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.737030    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcqgh\" (UniqueName: \"kubernetes.io/projected/928bd27a-d62d-4318-98a2-922d902f70ba-kube-api-access-tcqgh\") pod \"928bd27a-d62d-4318-98a2-922d902f70ba\" (UID: \"928bd27a-d62d-4318-98a2-922d902f70ba\") "
Aug 04 19:03:50.739261 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.739231    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928bd27a-d62d-4318-98a2-922d902f70ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "928bd27a-d62d-4318-98a2-922d902f70ba" (UID: "928bd27a-d62d-4318-98a2-922d902f70ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 19:03:50.739385 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.739325    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928bd27a-d62d-4318-98a2-922d902f70ba-kube-api-access-tcqgh" (OuterVolumeSpecName: "kube-api-access-tcqgh") pod "928bd27a-d62d-4318-98a2-922d902f70ba" (UID: "928bd27a-d62d-4318-98a2-922d902f70ba"). InnerVolumeSpecName "kube-api-access-tcqgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:03:50.838550 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.838456    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928bd27a-d62d-4318-98a2-922d902f70ba-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 19:03:50.838550 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.838490    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcqgh\" (UniqueName: \"kubernetes.io/projected/928bd27a-d62d-4318-98a2-922d902f70ba-kube-api-access-tcqgh\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\""
Aug 04 19:03:50.926366 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.926332    2576 generic.go:356] "Generic (PLEG): container finished" podID="928bd27a-d62d-4318-98a2-922d902f70ba" containerID="af404228a1d48eb0ab532d59f301685a2833d8a48af03e613016ad1730ff9d16" exitCode=0
Aug 04 19:03:50.926532 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.926372    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc" event={"ID":"928bd27a-d62d-4318-98a2-922d902f70ba","Type":"ContainerDied","Data":"af404228a1d48eb0ab532d59f301685a2833d8a48af03e613016ad1730ff9d16"}
Aug 04 19:03:50.926532 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.926380    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"
Aug 04 19:03:50.926532 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.926414    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc" event={"ID":"928bd27a-d62d-4318-98a2-922d902f70ba","Type":"ContainerDied","Data":"5c891cbb53eb39f82d8b91b75b1fc58667337c62986d5f602e404cd61c774df5"}
Aug 04 19:03:50.926532 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.926436    2576 scope.go:117] "RemoveContainer" containerID="af404228a1d48eb0ab532d59f301685a2833d8a48af03e613016ad1730ff9d16"
Aug 04 19:03:50.935363 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.935344    2576 scope.go:117] "RemoveContainer" containerID="af404228a1d48eb0ab532d59f301685a2833d8a48af03e613016ad1730ff9d16"
Aug 04 19:03:50.935613 ip-10-0-133-187 kubenswrapper[2576]: E0804 19:03:50.935592    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af404228a1d48eb0ab532d59f301685a2833d8a48af03e613016ad1730ff9d16\": container with ID starting with af404228a1d48eb0ab532d59f301685a2833d8a48af03e613016ad1730ff9d16 not found: ID does not exist" containerID="af404228a1d48eb0ab532d59f301685a2833d8a48af03e613016ad1730ff9d16"
Aug 04 19:03:50.935681 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.935619    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af404228a1d48eb0ab532d59f301685a2833d8a48af03e613016ad1730ff9d16"} err="failed to get container status \"af404228a1d48eb0ab532d59f301685a2833d8a48af03e613016ad1730ff9d16\": rpc error: code = NotFound desc = could not find container \"af404228a1d48eb0ab532d59f301685a2833d8a48af03e613016ad1730ff9d16\": container with ID starting with af404228a1d48eb0ab532d59f301685a2833d8a48af03e613016ad1730ff9d16 not found: ID does not exist"
Aug 04 19:03:50.948608 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.948581    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"]
Aug 04 19:03:50.952704 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:50.952680    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffcg2mc"]
Aug 04 19:03:52.582649 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:52.582616    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="928bd27a-d62d-4318-98a2-922d902f70ba" path="/var/lib/kubelet/pods/928bd27a-d62d-4318-98a2-922d902f70ba/volumes"
Aug 04 19:03:55.359264 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:55.359230    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:03:57.574961 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:03:57.574925    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:04:00.408765 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:04:00.408728    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:04:19.560919 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:04:19.560885    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:04:21.619805 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:04:21.619770    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:04:23.531336 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:04:23.531248    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:04:29.082897 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:04:29.082852    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:04:38.239529 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:04:38.239493    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:04:43.942639 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:04:43.942567    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:04:44.203735 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:04:44.203621    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:06:01.941084 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:06:01.941033    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:06:21.589259 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:06:21.589221    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:06:21.687253 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:06:21.687206    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:06:24.284332 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:06:24.284300    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:06:38.322781 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:06:38.322747    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:06:44.212473 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:06:44.212436    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:06:47.630622 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:06:47.630589    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:06:47.729462 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:06:47.729424    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:06:50.344651 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:06:50.344611    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:06:56.043322 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:06:56.043283    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:06:56.106566 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:06:56.106532    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:07:11.821543 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:11.821504    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-mm56x_577e1807-3f75-4898-8590-7edc7aca456a/authorino/0.log"
Aug 04 19:07:12.103041 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:12.102956    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-jjv5b_f7f3e0c7-3710-4b64-a1a0-d7a40c1f5fdd/manager/0.log"
Aug 04 19:07:16.031134 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:16.031069    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:07:16.234272 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:16.234234    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:07:16.369033 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:16.369000    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:07:19.041405 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:19.041370    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6kdvz_e9274e13-87e6-4bb8-bb29-3ed6fcde80e0/global-pull-secret-syncer/0.log"
Aug 04 19:07:19.140676 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:19.140644    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log"
Aug 04 19:07:19.238637 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:19.238603    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-187.ec2.internal_442faadf703363fe454f012df8df1128/haproxy/0.log"
Aug 04 19:07:21.952250 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:21.952209    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:07:22.053003 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:22.052970    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:07:24.044699 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:24.044659    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-mm56x_577e1807-3f75-4898-8590-7edc7aca456a/authorino/0.log"
Aug 04 19:07:24.390674 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:24.390643    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-jjv5b_f7f3e0c7-3710-4b64-a1a0-d7a40c1f5fdd/manager/0.log"
Aug 04 19:07:25.764193 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:25.764095    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d366eb8e-383b-43f3-93a9-c133cedf8334/alertmanager/0.log"
Aug 04 19:07:25.809028 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:25.809002    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d366eb8e-383b-43f3-93a9-c133cedf8334/config-reloader/0.log"
Aug 04 19:07:25.848947 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:25.848913    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d366eb8e-383b-43f3-93a9-c133cedf8334/kube-rbac-proxy-web/0.log"
Aug 04 19:07:25.892859 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:25.892821    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d366eb8e-383b-43f3-93a9-c133cedf8334/kube-rbac-proxy/0.log"
Aug 04 19:07:25.938937 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:25.938909    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d366eb8e-383b-43f3-93a9-c133cedf8334/kube-rbac-proxy-metric/0.log"
Aug 04 19:07:25.978827 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:25.978794    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d366eb8e-383b-43f3-93a9-c133cedf8334/prom-label-proxy/0.log"
Aug 04 19:07:26.027265 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:26.027190    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d366eb8e-383b-43f3-93a9-c133cedf8334/init-config-reloader/0.log"
Aug 04 19:07:26.130392 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:26.130361    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-bsx9g_a0ec7b7e-324b-4450-b564-78772ca59e95/kube-state-metrics/0.log"
Aug 04 19:07:26.168029 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:26.167997    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-bsx9g_a0ec7b7e-324b-4450-b564-78772ca59e95/kube-rbac-proxy-main/0.log"
Aug 04 19:07:26.213173 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:26.213144    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-bsx9g_a0ec7b7e-324b-4450-b564-78772ca59e95/kube-rbac-proxy-self/0.log"
Aug 04 19:07:26.264998 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:26.264967    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-678f56c7df-qv25h_79e7784b-6a83-4397-ad9e-aae932de61b9/metrics-server/0.log"
Aug 04 19:07:26.306973 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:26.306947    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-hcjnj_7383bd99-ba08-47db-b119-17066b86f867/monitoring-plugin/0.log"
Aug 04 19:07:26.666620 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:26.666547    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rrr8_7c755fd3-d8ce-4de8-9348-251d30952904/node-exporter/0.log"
Aug 04 19:07:26.703034 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:26.702994    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rrr8_7c755fd3-d8ce-4de8-9348-251d30952904/kube-rbac-proxy/0.log"
Aug 04 19:07:26.742820 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:26.742795    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rrr8_7c755fd3-d8ce-4de8-9348-251d30952904/init-textfile/0.log"
Aug 04 19:07:26.788858 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:26.788832    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-mw69r_c5bfce57-8a11-487c-8840-53bdd34ec60a/kube-rbac-proxy-main/0.log"
Aug 04 19:07:26.833547 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:26.833523    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-mw69r_c5bfce57-8a11-487c-8840-53bdd34ec60a/kube-rbac-proxy-self/0.log"
Aug 04 19:07:26.875433 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:26.875402    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-mw69r_c5bfce57-8a11-487c-8840-53bdd34ec60a/openshift-state-metrics/0.log"
Aug 04 19:07:26.944280 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:26.944206    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_163345cf-fd0e-4d1b-a2c4-aef129b9f525/prometheus/0.log"
Aug 04 19:07:26.992980 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:26.992950    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_163345cf-fd0e-4d1b-a2c4-aef129b9f525/config-reloader/0.log"
Aug 04 19:07:27.074019 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.073992    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_163345cf-fd0e-4d1b-a2c4-aef129b9f525/thanos-sidecar/0.log"
Aug 04 19:07:27.123033 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.123001    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_163345cf-fd0e-4d1b-a2c4-aef129b9f525/kube-rbac-proxy-web/0.log"
Aug 04 19:07:27.192987 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.192959    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_163345cf-fd0e-4d1b-a2c4-aef129b9f525/kube-rbac-proxy/0.log"
Aug 04 19:07:27.249837 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.249768    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_163345cf-fd0e-4d1b-a2c4-aef129b9f525/kube-rbac-proxy-thanos/0.log"
Aug 04 19:07:27.289868 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.289844    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_163345cf-fd0e-4d1b-a2c4-aef129b9f525/init-config-reloader/0.log"
Aug 04 19:07:27.333350 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.333321    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-frrgr_1a21274d-e369-4eb0-bc41-11d724999623/prometheus-operator/0.log"
Aug 04 19:07:27.356958 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.356917    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-frrgr_1a21274d-e369-4eb0-bc41-11d724999623/kube-rbac-proxy/0.log"
Aug 04 19:07:27.387460 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.387435    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-fh84v_f220e635-729a-4535-aa7a-27ffe2bc2fb1/prometheus-operator-admission-webhook/0.log"
Aug 04 19:07:27.440354 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.440327    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c5d56dc58-5h4n6_f2492582-86ad-425c-9204-06faf9569f4b/telemeter-client/0.log"
Aug 04 19:07:27.483974 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.483945    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c5d56dc58-5h4n6_f2492582-86ad-425c-9204-06faf9569f4b/reload/0.log"
Aug 04 19:07:27.521916 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.521840    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c5d56dc58-5h4n6_f2492582-86ad-425c-9204-06faf9569f4b/kube-rbac-proxy/0.log"
Aug 04 19:07:27.571479 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.571449    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c679f8c78-2rshq_8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4/thanos-query/0.log"
Aug 04 19:07:27.606876 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.606847    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c679f8c78-2rshq_8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4/kube-rbac-proxy-web/0.log"
Aug 04 19:07:27.645769 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.645745    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c679f8c78-2rshq_8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4/kube-rbac-proxy/0.log"
Aug 04 19:07:27.690733 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.690709    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c679f8c78-2rshq_8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4/prom-label-proxy/0.log"
Aug 04 19:07:27.731784 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.731760    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c679f8c78-2rshq_8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4/kube-rbac-proxy-rules/0.log"
Aug 04 19:07:27.777866 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:27.777796    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c679f8c78-2rshq_8222a7d2-3a06-4ae1-9f5a-9ccfc9da28d4/kube-rbac-proxy-metrics/0.log"
Aug 04 19:07:29.023794 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.023753    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"]
Aug 04 19:07:29.024218 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.024150    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="928bd27a-d62d-4318-98a2-922d902f70ba" containerName="main"
Aug 04 19:07:29.024218 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.024165    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="928bd27a-d62d-4318-98a2-922d902f70ba" containerName="main"
Aug 04 19:07:29.024218 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.024179    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2dde748-3196-4303-b1f3-5b64c4b2b59a" containerName="main"
Aug 04 19:07:29.024218 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.024185    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2dde748-3196-4303-b1f3-5b64c4b2b59a" containerName="main"
Aug 04 19:07:29.024218 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.024206    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba956419-88c0-4184-bfc4-f44283b3745d" containerName="main"
Aug 04 19:07:29.024218 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.024213    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba956419-88c0-4184-bfc4-f44283b3745d" containerName="main"
Aug 04 19:07:29.024426 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.024269    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2dde748-3196-4303-b1f3-5b64c4b2b59a" containerName="main"
Aug 04 19:07:29.024426 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.024278    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="928bd27a-d62d-4318-98a2-922d902f70ba" containerName="main"
Aug 04 19:07:29.024426 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.024284    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba956419-88c0-4184-bfc4-f44283b3745d" containerName="main"
Aug 04 19:07:29.027620 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.027598    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.029874 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.029854    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m6lzs\"/\"openshift-service-ca.crt\""
Aug 04 19:07:29.030771 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.030750    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m6lzs\"/\"default-dockercfg-jqf5h\""
Aug 04 19:07:29.030771 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.030757    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m6lzs\"/\"kube-root-ca.crt\""
Aug 04 19:07:29.034997 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.034754    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"]
Aug 04 19:07:29.188628 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.188591    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3df9719f-1e45-4512-ac59-a4422d28fc46-proc\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.188808 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.188645    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3df9719f-1e45-4512-ac59-a4422d28fc46-podres\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.188808 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.188687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rp2\" (UniqueName: \"kubernetes.io/projected/3df9719f-1e45-4512-ac59-a4422d28fc46-kube-api-access-g5rp2\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.188808 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.188760    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3df9719f-1e45-4512-ac59-a4422d28fc46-lib-modules\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.188808 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.188780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3df9719f-1e45-4512-ac59-a4422d28fc46-sys\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.289244 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.289215    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3df9719f-1e45-4512-ac59-a4422d28fc46-lib-modules\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.289405 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.289252    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3df9719f-1e45-4512-ac59-a4422d28fc46-sys\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.289405 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.289308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3df9719f-1e45-4512-ac59-a4422d28fc46-proc\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.289405 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.289347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3df9719f-1e45-4512-ac59-a4422d28fc46-podres\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.289554 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.289414    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3df9719f-1e45-4512-ac59-a4422d28fc46-proc\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.289554 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.289414    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3df9719f-1e45-4512-ac59-a4422d28fc46-lib-modules\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.289554 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.289416    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3df9719f-1e45-4512-ac59-a4422d28fc46-sys\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.289554 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.289435    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rp2\" (UniqueName: \"kubernetes.io/projected/3df9719f-1e45-4512-ac59-a4422d28fc46-kube-api-access-g5rp2\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.289554 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.289517    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3df9719f-1e45-4512-ac59-a4422d28fc46-podres\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.297450 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.297424    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rp2\" (UniqueName: \"kubernetes.io/projected/3df9719f-1e45-4512-ac59-a4422d28fc46-kube-api-access-g5rp2\") pod \"perf-node-gather-daemonset-lpwr9\" (UID: \"3df9719f-1e45-4512-ac59-a4422d28fc46\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.338373 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.338349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.466212 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.466183    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"]
Aug 04 19:07:29.468731 ip-10-0-133-187 kubenswrapper[2576]: W0804 19:07:29.468697    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3df9719f_1e45_4512_ac59_a4422d28fc46.slice/crio-cda02613d414a4ca103d40ef8e488098e67bf2b9a5e63bb5459e43d472dc5073 WatchSource:0}: Error finding container cda02613d414a4ca103d40ef8e488098e67bf2b9a5e63bb5459e43d472dc5073: Status 404 returned error can't find the container with id cda02613d414a4ca103d40ef8e488098e67bf2b9a5e63bb5459e43d472dc5073
Aug 04 19:07:29.942653 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.942562    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9" event={"ID":"3df9719f-1e45-4512-ac59-a4422d28fc46","Type":"ContainerStarted","Data":"9c9327ab47ac2a2fd4b6254c0513cd121a324c70a0a853610eb6a7cb0181f750"}
Aug 04 19:07:29.942653 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.942610    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9" event={"ID":"3df9719f-1e45-4512-ac59-a4422d28fc46","Type":"ContainerStarted","Data":"cda02613d414a4ca103d40ef8e488098e67bf2b9a5e63bb5459e43d472dc5073"}
Aug 04 19:07:29.942653 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.942646    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:29.959019 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:29.958970    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9" podStartSLOduration=0.958955474 podStartE2EDuration="958.955474ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:07:29.95799777 +0000 UTC m=+4049.933829461" watchObservedRunningTime="2026-08-04 19:07:29.958955474 +0000 UTC m=+4049.934787170"
Aug 04 19:07:31.452082 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:31.452053    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w7w5n_cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a/dns/0.log"
Aug 04 19:07:31.486861 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:31.486835    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w7w5n_cdd43be5-2ebd-4f8d-acc2-2f990ee26f6a/kube-rbac-proxy/0.log"
Aug 04 19:07:31.560934 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:31.560906    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vs6m9_924363b2-77bd-4de5-9279-8fdf18129796/dns-node-resolver/0.log"
Aug 04 19:07:32.163521 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:32.163482    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t69hm_22866d7a-8cbe-4c71-8ffb-507f5d4b879f/node-ca/0.log"
Aug 04 19:07:33.918317 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:33.918288    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-27kng_3155211c-54d6-4956-9506-85213a63de71/serve-healthcheck-canary/0.log"
Aug 04 19:07:34.526864 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:34.526834    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4m69_25b94f13-bec2-41c1-a848-37a3d18170a9/kube-rbac-proxy/0.log"
Aug 04 19:07:34.550592 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:34.550566    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4m69_25b94f13-bec2-41c1-a848-37a3d18170a9/exporter/0.log"
Aug 04 19:07:34.573341 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:34.573308    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4m69_25b94f13-bec2-41c1-a848-37a3d18170a9/extractor/0.log"
Aug 04 19:07:35.120838 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:35.120804    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ggjx_b7c40a2d-3dd2-4a5b-95f4-e8621ca93c30/konnectivity-agent/0.log" limit=1024
Aug 04 19:07:35.958274 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:35.958244    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-lpwr9"
Aug 04 19:07:37.661305 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:37.661277    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68dff9b49f-ztkcf_7fc7c669-43c4-4ade-8f8f-1c7327792580/manager/0.log"
Aug 04 19:07:37.692361 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:37.692333    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-q8dkb_742530c4-d506-4393-a9e7-709de7b8580e/openshift-lws-operator/0.log"
Aug 04 19:07:38.840058 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:38.840027    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-5db69799c8-6bpxm_97ad7257-8e03-4558-bd98-d0a752177a5f/manager/0.log"
Aug 04 19:07:39.675060 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:39.674971    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-wqxwt_376dbb1f-8af7-4243-b342-2a20d20418db/manager/0.log"
Aug 04 19:07:45.791349 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:45.791317    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5q6q7_f0d65e6e-91ca-476e-b30e-3da05d33616e/kube-multus/0.log"
Aug 04 19:07:46.178720 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:46.178640    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zh6dz_707bcbec-504f-4e07-ae6f-6d6b4d16964e/kube-multus-additional-cni-plugins/0.log"
Aug 04 19:07:46.201693 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:46.201667    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zh6dz_707bcbec-504f-4e07-ae6f-6d6b4d16964e/egress-router-binary-copy/0.log"
Aug 04 19:07:46.225506 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:46.225474    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zh6dz_707bcbec-504f-4e07-ae6f-6d6b4d16964e/cni-plugins/0.log"
Aug 04 19:07:46.249913 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:46.249881    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zh6dz_707bcbec-504f-4e07-ae6f-6d6b4d16964e/bond-cni-plugin/0.log"
Aug 04 19:07:46.278002 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:46.277975    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zh6dz_707bcbec-504f-4e07-ae6f-6d6b4d16964e/routeoverride-cni/0.log"
Aug 04 19:07:46.301619 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:46.301590    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zh6dz_707bcbec-504f-4e07-ae6f-6d6b4d16964e/whereabouts-cni-bincopy/0.log"
Aug 04 19:07:46.325874 ip-10-0-133-187 kubenswrapper[2576]: I0804 19:07:46.325849    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zh6dz_707bcbec-504f-4e07-ae6f-6d6b4d16964e/whereabouts-cni/0.log"