Sep 09 03:58:20.288704 ip-10-0-135-219 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 03:58:20.757869 ip-10-0-135-219 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:58:20.757869 ip-10-0-135-219 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 03:58:20.757869 ip-10-0-135-219 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:58:20.757869 ip-10-0-135-219 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 03:58:20.758407 ip-10-0-135-219 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:58:20.758407 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.757939    2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 03:58:20.760822 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760806    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:58:20.760822 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760822    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760826    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760830    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760835    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760839    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760843    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760846    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760850    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760853    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760855    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760863    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760865    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760868    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760871    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760874    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760876    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760879    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760882    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760885    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:58:20.760883 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760888    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760891    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760894    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760897    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760899    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760902    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760906    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760910    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760913    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760916    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760919    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760921    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760924    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760927    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760929    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760932    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760934    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760936    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760939    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760941    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:58:20.761332 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760944    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760946    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760948    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760951    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760953    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760955    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760958    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760960    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760962    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760965    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760967    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760969    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760972    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760974    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760977    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760980    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760982    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760985    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760987    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760990    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:58:20.761838 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760992    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760994    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760997    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.760999    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761002    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761004    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761006    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761009    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761011    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761014    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761016    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761019    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761021    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761025    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761028    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761030    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761033    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761035    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761037    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761040    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:58:20.762309 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761042    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761045    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761047    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761050    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761052    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761055    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761057    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761060    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761062    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761065    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761067    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761070    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761072    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761075    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761077    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761079    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761082    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761084    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:58:20.762780 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.761086    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763167    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763175    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763179    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763182    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763186    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763190    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763193    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763196    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763199    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763202    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763204    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763207    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763210    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763212    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763215    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763218    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763220    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763223    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:58:20.763259 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763226    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763230    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763233    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763235    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763237    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763240    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763242    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763245    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763248    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763250    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763253    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763255    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763257    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763260    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763263    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763265    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763268    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763271    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763273    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:58:20.763761 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763276    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763279    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763282    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763284    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763286    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763289    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763291    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763293    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763296    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763299    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763301    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763305    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763308    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763311    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763314    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763317    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763320    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763323    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763325    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763327    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:58:20.764244 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763331    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763333    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763336    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763338    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763341    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763343    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763346    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763349    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763352    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763354    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763356    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763359    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763361    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763364    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763366    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763369    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763371    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763374    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763376    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:58:20.764739 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763378    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763381    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763383    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763385    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763388    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763391    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763393    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763395    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763398    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763401    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763403    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763406    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763408    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763411    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763413    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763416    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763418    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763421    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763423    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763426    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:58:20.765194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763428    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763430    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763433    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763789    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763794    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763797    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763800    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763803    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763806    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763809    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763811    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763814    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763816    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763819    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763821    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763824    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763828    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763831    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763834    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763837    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:58:20.765672 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763840    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763843    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763845    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763848    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763850    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763853    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763856    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763858    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763861    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763863    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763866    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763868    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763871    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763873    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763876    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763878    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763881    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763884    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763886    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763889    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:58:20.766139 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763892    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763894    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763897    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763899    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763910    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763914    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763917    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763920    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763922    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763925    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763928    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763930    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763933    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763936    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763938    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763941    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763945    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763948    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763951    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:58:20.766618 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763954    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763969    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763971    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763974    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763976    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763978    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763981    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763984    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763987    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763989    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763992    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763994    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763996    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.763999    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764001    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764004    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764006    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764008    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764011    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:58:20.767090 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764013    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764015    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764017    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764020    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764022    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764024    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764026    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764029    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764031    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764033    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764036    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764040    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764043    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764046    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764049    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764052    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764054    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764056    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764058    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764061    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:58:20.767577 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764063    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764066    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764068    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764070    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764134    2553 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764166    2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764171    2553 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764175    2553 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764179    2553 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764183    2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764186    2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764190    2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764193    2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764196    2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764199    2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764202    2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764205    2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764208    2553 flags.go:64] FLAG: --cgroup-root=""
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764210    2553 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764213    2553 flags.go:64] FLAG: --client-ca-file=""
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764215    2553 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764218    2553 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764222    2553 flags.go:64] FLAG: --cluster-domain=""
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764225    2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 03:58:20.768114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764228    2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764231    2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764234    2553 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764237    2553 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764240    2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764243    2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764246    2553 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764249    2553 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764251    2553 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764254    2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764257    2553 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764260    2553 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764264    2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764266    2553 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764269    2553 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764272    2553 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764274    2553 flags.go:64] FLAG: --enable-server="true"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764280    2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764284    2553 flags.go:64] FLAG: --event-burst="100"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764286    2553 flags.go:64] FLAG: --event-qps="50"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764289    2553 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764292    2553 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764295    2553 flags.go:64] FLAG: --eviction-hard=""
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764298    2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 03:58:20.768685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764300    2553 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764303    2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764306    2553 flags.go:64] FLAG: --eviction-soft=""
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764308    2553 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764311    2553 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764314    2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764318    2553 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764321    2553 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764324    2553 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764326    2553 flags.go:64] FLAG: --feature-gates=""
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764330    2553 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764333    2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764335    2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764338    2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764341    2553 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764343    2553 flags.go:64] FLAG: --help="false"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764346    2553 flags.go:64] FLAG: --hostname-override="ip-10-0-135-219.ec2.internal"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764349    2553 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764352    2553 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764354    2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764357    2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764360    2553 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764363    2553 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 03:58:20.769277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764365    2553 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764368    2553 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764371    2553 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764374    2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764377    2553 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764379    2553 flags.go:64] FLAG: --kube-reserved=""
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764382    2553 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764385    2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764387    2553 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764390    2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764392    2553 flags.go:64] FLAG: --lock-file=""
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764395    2553 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764397    2553 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764400    2553 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764405    2553 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764409    2553 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764412    2553 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764415    2553 flags.go:64] FLAG: --logging-format="text"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764417    2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764420    2553 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764423    2553 flags.go:64] FLAG: --manifest-url=""
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764425    2553 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764430    2553 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764432    2553 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764436    2553 flags.go:64] FLAG: --max-pods="110"
Sep 09 03:58:20.769865 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764452    2553 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764455    2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764458    2553 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764461    2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764464    2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764467    2553 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764469    2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764476    2553 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764479    2553 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764482    2553 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764485    2553 flags.go:64] FLAG: --pod-cidr=""
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764488    2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764492    2553 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764495    2553 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764498    2553 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764501    2553 flags.go:64] FLAG: --port="10250"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764504    2553 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764506    2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06e877bcbf3022ae1"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764509    2553 flags.go:64] FLAG: --qos-reserved=""
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764512    2553 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764514    2553 flags.go:64] FLAG: --register-node="true"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764517    2553 flags.go:64] FLAG: --register-with-taints=""
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764520    2553 flags.go:64] FLAG: --registry-burst="10"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764526    2553 flags.go:64] FLAG: --registry-qps="5"
Sep 09 03:58:20.770566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764529    2553 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764531    2553 flags.go:64] FLAG: --reserved-memory=""
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764535    2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764537    2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764540    2553 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764543    2553 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764545    2553 flags.go:64] FLAG: --runonce="false"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764548    2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764551    2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764554    2553 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764556    2553 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764559    2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764561    2553 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764564    2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764567    2553 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764569    2553 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764572    2553 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764574    2553 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764577    2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764580    2553 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764583    2553 flags.go:64] FLAG: --system-cgroups=""
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764585    2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764591    2553 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764593    2553 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764596    2553 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 03:58:20.771149 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764600    2553 flags.go:64] FLAG: --tls-min-version=""
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764602    2553 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764605    2553 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764607    2553 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764610    2553 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764612    2553 flags.go:64] FLAG: --v="2"
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764617    2553 flags.go:64] FLAG: --version="false"
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764621    2553 flags.go:64] FLAG: --vmodule=""
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764624    2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.764627    2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764750    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764754    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764757    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764760    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764763    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764765    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764768    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764770    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764773    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764775    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764778    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764780    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:58:20.771832 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764783    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764785    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764788    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764790    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764793    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764797    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764800    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764803    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764806    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764809    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764812    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764815    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764818    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764821    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764823    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764825    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764832    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764834    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:58:20.772364 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764837    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764839    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764841    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764844    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764847    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764851    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764854    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764857    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764859    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764862    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764864    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764866    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764869    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764871    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764874    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764876    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764878    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764881    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764883    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764885    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:58:20.772856 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764888    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764890    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764892    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764895    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764897    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764900    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764902    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764904    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764906    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764909    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764915    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764917    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764920    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764922    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764924    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764926    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764929    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764931    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764934    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764936    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:58:20.773321 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764938    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764941    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764943    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764946    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764948    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764950    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764952    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764955    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764957    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764959    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764962    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764964    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764966    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764969    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764971    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764974    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764976    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764979    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764981    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764983    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:58:20.773803 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764986    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764988    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764993    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764996    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.764998    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.765001    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.765003    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.765005    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.765008    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.765658    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.772735    2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.772749    2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.772766    2553 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.772771    2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772829    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:58:20.774264 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772834    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772837    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772840    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772844    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772847    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772850    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772853    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772855    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772858    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772862    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772866    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772874    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772878    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772880    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772883    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772886    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772888    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772896    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772900    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:58:20.774657 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772903    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772906    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772908    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772911    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772913    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772916    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772918    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772920    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772923    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772926    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772928    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772931    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772933    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772935    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772938    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772940    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772942    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772944    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772947    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772949    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:58:20.775172 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772951    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772954    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772956    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772958    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772960    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772970    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772973    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772976    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772979    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772982    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772984    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772986    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772988    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772991    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772993    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772995    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.772998    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773000    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773002    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773005    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:58:20.775669 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773008    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773010    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773013    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773015    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773018    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773020    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773022    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773025    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773027    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773030    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773032    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773034    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773036    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773039    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773041    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773043    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773045    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773048    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773058    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773060    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:58:20.776206 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773063    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773065    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773067    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773070    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773072    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773074    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773077    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773079    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773081    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773084    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773086    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773089    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773091    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773094    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773096    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773098    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773101    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773103    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:58:20.776751 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773105    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.773110    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773251    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773256    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773258    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773261    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773263    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773266    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773268    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773270    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773273    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773276    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773278    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773286    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773288    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:58:20.777311 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773290    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773293    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773295    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773297    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773300    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773302    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773306    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773308    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773311    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773313    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773316    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773318    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773320    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773323    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773325    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773328    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773330    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773332    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773335    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:58:20.777701 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773338    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773340    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773342    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773345    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773347    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773349    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773352    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773354    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773356    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773359    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773362    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773366    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773374    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773376    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773379    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773382    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773384    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773387    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773389    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773392    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:58:20.778348 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773395    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773397    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773399    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773402    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773404    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773407    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773410    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773413    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773415    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773418    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773420    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773423    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773426    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773428    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773430    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773433    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773435    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773437    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773453    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773455    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:58:20.778990 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773458    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773460    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773463    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773465    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773467    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773474    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773477    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773479    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773482    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773484    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773486    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773489    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773491    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773493    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773496    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773498    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773500    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773503    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773505    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:58:20.779775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773507    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:58:20.780352 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773510    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:58:20.780352 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773512    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:58:20.780352 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773514    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:58:20.780352 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773516    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:58:20.780352 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773519    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:58:20.780352 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773521    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:58:20.780352 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:20.773523    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:58:20.780352 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.773528    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:58:20.780352 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.773635    2553 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 03:58:20.780352 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.775794    2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 03:58:20.780352 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.776758    2553 server.go:1019] "Starting client certificate rotation"
Sep 09 03:58:20.780352 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.776875    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 03:58:20.780352 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.776979    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 03:58:20.781019 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.781004    2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 03:58:20.782130 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.782112    2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 03:58:20.791279 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.791264    2553 log.go:25] "Validated CRI v1 runtime API"
Sep 09 03:58:20.797276 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.797261    2553 log.go:25] "Validated CRI v1 image API"
Sep 09 03:58:20.798140 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.798117    2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 03:58:20.802177 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.802153    2553 fs.go:135] Filesystem UUIDs: map[29d6b683-0493-4e1c-990f-c33a0b3c7ab3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d5b0ac7c-2d84-4e4b-9b0e-ba468951bdba:/dev/nvme0n1p4]
Sep 09 03:58:20.802244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.802176    2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 03:58:20.808148 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.808049    2553 manager.go:216] Machine: {Timestamp:2026-09-09 03:58:20.806613875 +0000 UTC m=+0.332984596 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3186533 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec261d4294014f4466a9eec27c4dbc74 SystemUUID:ec261d42-9401-4f44-66a9-eec27c4dbc74 BootID:d231a278-81b7-4e0b-af62-50b7e2d3ad46 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d4:f7:f9:ae:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d4:f7:f9:ae:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:d9:5f:10:59:fc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 03:58:20.808148 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.808145    2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 03:58:20.808258 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.808237    2553 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 03:58:20.808863 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.808839    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 03:58:20.811308 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.811283    2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 03:58:20.811457 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.811311    2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-219.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 03:58:20.812726 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.812714    2553 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 03:58:20.812762 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.812729    2553 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 03:58:20.812762 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.812736    2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 03:58:20.813511 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.813501    2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 03:58:20.815107 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.815098    2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 03:58:20.820098 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.820088    2553 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 03:58:20.820208 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.820199    2553 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 03:58:20.820253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.820245    2553 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 03:58:20.820281 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.820265    2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 03:58:20.820281 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.820278    2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 03:58:20.820355 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.820287    2553 kubelet.go:398] "Adding apiserver pod source"
Sep 09 03:58:20.820355 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.820300    2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 03:58:20.820811 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.820797    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 03:58:20.820861 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.820821    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 03:58:20.824159 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.823154    2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 03:58:20.826030 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.826012    2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 03:58:20.826119 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.826048    2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 03:58:20.826190 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.826180    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 03:58:20.826226 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.826195    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 03:58:20.826226 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.826203    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 03:58:20.826226 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.826211    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 03:58:20.826226 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.826220    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 03:58:20.826226 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.826225    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 03:58:20.826924 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.826909    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 03:58:20.826924 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.826922    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 03:58:20.826992 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.826929    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 03:58:20.826992 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.826934    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 03:58:20.826992 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.826942    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 03:58:20.826992 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.826949    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 03:58:20.827797 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.827786    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 03:58:20.827797 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.827797    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 03:58:20.830144 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.830125    2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-219.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 03:58:20.830401 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:20.830376    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 03:58:20.830549 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:20.830530    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 03:58:20.831400 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.831387    2553 server.go:1268] "Started kubelet"
Sep 09 03:58:20.831520 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.831484    2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 03:58:20.831629 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.831537    2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 03:58:20.831629 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.831613    2553 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 03:58:20.832196 ip-10-0-135-219 systemd[1]: Started Kubernetes Kubelet.
Sep 09 03:58:20.832856 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.832802    2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 03:58:20.833998 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.833983    2553 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 03:58:20.834491 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.834410    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jtplp"
Sep 09 03:58:20.836802 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:20.836783    2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 03:58:20.837221 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:20.836194    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-219.ec2.internal.18d38afe126832c3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-219.ec2.internal,UID:ip-10-0-135-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-219.ec2.internal,},FirstTimestamp:2026-09-09 03:58:20.831363779 +0000 UTC m=+0.357734480,LastTimestamp:2026-09-09 03:58:20.831363779 +0000 UTC m=+0.357734480,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-219.ec2.internal,}"
Sep 09 03:58:20.838126 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.838108    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 03:58:20.838673 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.838657    2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 03:58:20.838759 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:20.838740    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:20.839259 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.839240    2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 03:58:20.839361 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.839349    2553 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 03:58:20.839361 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.839361    2553 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 03:58:20.839520 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.839502    2553 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 03:58:20.839520 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.839508    2553 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 03:58:20.839623 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.839539    2553 factory.go:55] Registering systemd factory
Sep 09 03:58:20.839623 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.839554    2553 factory.go:223] Registration of the systemd container factory successfully
Sep 09 03:58:20.840015 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.839994    2553 factory.go:153] Registering CRI-O factory
Sep 09 03:58:20.840015 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.840011    2553 factory.go:223] Registration of the crio container factory successfully
Sep 09 03:58:20.840114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.840054    2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 03:58:20.840114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.840066    2553 factory.go:103] Registering Raw factory
Sep 09 03:58:20.840114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.840089    2553 manager.go:1196] Started watching for new ooms in manager
Sep 09 03:58:20.840416 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.840406    2553 manager.go:349] Starting recovery of all containers
Sep 09 03:58:20.842547 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.842525    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jtplp"
Sep 09 03:58:20.848586 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.848549    2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:58:20.851647 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.851628    2553 manager.go:354] Recovery completed
Sep 09 03:58:20.853822 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:20.853804    2553 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 09 03:58:20.856644 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.856630    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:58:20.857941 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:20.857922    2553 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-219.ec2.internal\" not found" node="ip-10-0-135-219.ec2.internal"
Sep 09 03:58:20.858117 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.858104    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:58:20.858175 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.858136    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:58:20.858175 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.858145    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:58:20.858716 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.858704    2553 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 03:58:20.858716 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.858715    2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 03:58:20.858844 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.858730    2553 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 03:58:20.858844 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.858768    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:58:20.859822 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.859807    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:58:20.859871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.859837    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:58:20.859871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.859851    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:58:20.860769 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.860757    2553 policy_none.go:49] "None policy: Start"
Sep 09 03:58:20.860801 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.860774    2553 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 03:58:20.860801 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.860784    2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 03:58:20.862659 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.862648    2553 policy_none.go:47] "Start"
Sep 09 03:58:20.896131 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.896033    2553 manager.go:337] "Starting Device Plugin manager"
Sep 09 03:58:20.896204 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:20.896152    2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 03:58:20.896204 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.896165    2553 server.go:87] "Starting device plugin registration server"
Sep 09 03:58:20.896426 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.896415    2553 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 03:58:20.896512 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.896427    2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 03:58:20.897324 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.897205    2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 03:58:20.897324 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:20.897256    2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 03:58:20.897324 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.897282    2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 03:58:20.897324 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.897292    2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 03:58:20.897324 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:20.897306    2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:20.980745 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.980719    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 03:58:20.982798 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.981992    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 03:58:20.982798 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.982040    2553 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 03:58:20.982798 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.982063    2553 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 03:58:20.982798 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:20.982130    2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 03:58:20.984168 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.984151    2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:58:20.997134 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.997121    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:58:20.997867 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.997855    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:58:20.997922 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.997881    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:58:20.997922 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.997890    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:58:20.997922 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:20.997908    2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.006581 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.006562    2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.006666 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:21.006586    2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-219.ec2.internal\": node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:21.035900 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:21.035846    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:21.083170 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.083152    2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-219.ec2.internal"]
Sep 09 03:58:21.083233 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.083206    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:58:21.084470 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.084437    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:58:21.084538 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.084484    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:58:21.084538 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.084494    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:58:21.086659 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.086647    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:58:21.086795 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.086770    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.086903 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.086813    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:58:21.087340 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.087326    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:58:21.087415 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.087350    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:58:21.087415 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.087369    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:58:21.087415 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.087378    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:58:21.087415 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.087352    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:58:21.087562 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.087455    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:58:21.089945 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.089932    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.089989 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.089960    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:58:21.090632 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.090617    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:58:21.090689 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.090649    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:58:21.090689 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.090664    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:58:21.112985 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:21.112963    2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-219.ec2.internal\" not found" node="ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.116601 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:21.116584    2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-219.ec2.internal\" not found" node="ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.136370 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:21.136352    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:21.140711 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.140695    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b2974e26bc83af5f46cffe86e53f1fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal\" (UID: \"3b2974e26bc83af5f46cffe86e53f1fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.140794 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.140723    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b2974e26bc83af5f46cffe86e53f1fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal\" (UID: \"3b2974e26bc83af5f46cffe86e53f1fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.140794 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.140751    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cbec884f0899e64243f4816842f939e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-219.ec2.internal\" (UID: \"5cbec884f0899e64243f4816842f939e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.236970 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:21.236951    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:21.241282 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.241269    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b2974e26bc83af5f46cffe86e53f1fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal\" (UID: \"3b2974e26bc83af5f46cffe86e53f1fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.241340 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.241305    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b2974e26bc83af5f46cffe86e53f1fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal\" (UID: \"3b2974e26bc83af5f46cffe86e53f1fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.241379 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.241342    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b2974e26bc83af5f46cffe86e53f1fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal\" (UID: \"3b2974e26bc83af5f46cffe86e53f1fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.241379 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.241361    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cbec884f0899e64243f4816842f939e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-219.ec2.internal\" (UID: \"5cbec884f0899e64243f4816842f939e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.241438 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.241379    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cbec884f0899e64243f4816842f939e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-219.ec2.internal\" (UID: \"5cbec884f0899e64243f4816842f939e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.241438 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.241399    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b2974e26bc83af5f46cffe86e53f1fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal\" (UID: \"3b2974e26bc83af5f46cffe86e53f1fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.337729 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:21.337670    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:21.416119 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.416103    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.419825 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.419808    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:21.438018 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:21.438001    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:21.538482 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:21.538463    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:21.638945 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:21.638885    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:21.739266 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:21.739243    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:21.776635 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.776606    2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 03:58:21.777087 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.776766    2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 03:58:21.777143 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.776730    2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 03:58:21.839494 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:21.839473    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:21.845694 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.845669    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 03:53:21 +0000 UTC" deadline="2028-05-29 04:07:58.236965058 +0000 UTC"
Sep 09 03:58:21.845694 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:21.845694    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15072h9m36.391273969s"
Sep 09 03:58:21.940030 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:21.940007    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:22.025142 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:22.025110    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2974e26bc83af5f46cffe86e53f1fa.slice/crio-889f2a4162528c098da6e6c45e543607469bc1e6d0e2ea309535ab59fc1bc0dc WatchSource:0}: Error finding container 889f2a4162528c098da6e6c45e543607469bc1e6d0e2ea309535ab59fc1bc0dc: Status 404 returned error can't find the container with id 889f2a4162528c098da6e6c45e543607469bc1e6d0e2ea309535ab59fc1bc0dc
Sep 09 03:58:22.029258 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.029240    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:58:22.040856 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:22.040837    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:22.089871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.089844    2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:58:22.141186 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:22.141161    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:22.241694 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:22.241626    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:22.342083 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:22.342059    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-219.ec2.internal\" not found"
Sep 09 03:58:22.343002 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.342989    2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:58:22.439092 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.439066    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:22.451013 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.450994    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:58:22.452024 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.452013    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-219.ec2.internal"
Sep 09 03:58:22.460805 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.460786    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:58:22.583643 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.583586    2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:58:22.821559 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.821532    2553 apiserver.go:52] "Watching apiserver"
Sep 09 03:58:22.831249 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.831219    2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 03:58:22.833955 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.833879    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kpwhr","kube-system/kube-apiserver-proxy-ip-10-0-135-219.ec2.internal","openshift-dns/node-resolver-86q49","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal","openshift-network-operator/iptables-alerter-xf8kb","openshift-ovn-kubernetes/ovnkube-node-v45h8","kube-system/konnectivity-agent-5sc28","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b","openshift-cluster-node-tuning-operator/tuned-jdmwv","openshift-image-registry/node-ca-k29sf","openshift-multus/multus-additional-cni-plugins-pj8mr","openshift-multus/multus-z5ftt","openshift-multus/network-metrics-daemon-4x7vw","openshift-network-diagnostics/network-check-target-njsng"]
Sep 09 03:58:22.836889 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.836865    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k29sf"
Sep 09 03:58:22.840101 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.839268    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:22.840101 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:22.839337    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:22.840101 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.840006    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r9xnc\""
Sep 09 03:58:22.840244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.840188    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 03:58:22.840362 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.840320    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 03:58:22.840628 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.840610    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 03:58:22.841047 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.841032    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 03:58:22.843413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.843392    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-86q49"
Sep 09 03:58:22.845447 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.845425    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:22.845563 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:22.845514    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:22.845935 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.845914    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 03:58:22.846118 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.846106    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jjf5g\""
Sep 09 03:58:22.846420 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.846399    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 03:58:22.847718 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.847692    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/18092042-a374-4054-9dfa-6b0a511d7c0f-hosts-file\") pod \"node-resolver-86q49\" (UID: \"18092042-a374-4054-9dfa-6b0a511d7c0f\") " pod="openshift-dns/node-resolver-86q49"
Sep 09 03:58:22.847819 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.847731    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18092042-a374-4054-9dfa-6b0a511d7c0f-tmp-dir\") pod \"node-resolver-86q49\" (UID: \"18092042-a374-4054-9dfa-6b0a511d7c0f\") " pod="openshift-dns/node-resolver-86q49"
Sep 09 03:58:22.847819 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.847757    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsr9n\" (UniqueName: \"kubernetes.io/projected/18092042-a374-4054-9dfa-6b0a511d7c0f-kube-api-access-tsr9n\") pod \"node-resolver-86q49\" (UID: \"18092042-a374-4054-9dfa-6b0a511d7c0f\") " pod="openshift-dns/node-resolver-86q49"
Sep 09 03:58:22.847819 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.847781    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67c61063-131e-4cc1-84ad-889cded7b195-host\") pod \"node-ca-k29sf\" (UID: \"67c61063-131e-4cc1-84ad-889cded7b195\") " pod="openshift-image-registry/node-ca-k29sf"
Sep 09 03:58:22.847819 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.847805    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8pfl\" (UniqueName: \"kubernetes.io/projected/67c61063-131e-4cc1-84ad-889cded7b195-kube-api-access-m8pfl\") pod \"node-ca-k29sf\" (UID: \"67c61063-131e-4cc1-84ad-889cded7b195\") " pod="openshift-image-registry/node-ca-k29sf"
Sep 09 03:58:22.848024 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.847830    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs\") pod \"network-metrics-daemon-4x7vw\" (UID: \"9579406a-2f97-456a-8d1a-bb8db3d24ab6\") " pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:22.848024 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.847855    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6fz\" (UniqueName: \"kubernetes.io/projected/9579406a-2f97-456a-8d1a-bb8db3d24ab6-kube-api-access-dd6fz\") pod \"network-metrics-daemon-4x7vw\" (UID: \"9579406a-2f97-456a-8d1a-bb8db3d24ab6\") " pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:22.848024 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.847879    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67c61063-131e-4cc1-84ad-889cded7b195-serviceca\") pod \"node-ca-k29sf\" (UID: \"67c61063-131e-4cc1-84ad-889cded7b195\") " pod="openshift-image-registry/node-ca-k29sf"
Sep 09 03:58:22.848180 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.848150    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xf8kb"
Sep 09 03:58:22.851110 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.851091    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.851257 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.851225    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7zlmw\""
Sep 09 03:58:22.851801 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.851552    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 03:58:22.851897 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.851810    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:58:22.851897 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.851834    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 03:58:22.853613 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.853595    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.853980 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.853960    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 03:58:22.854348 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.854323    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 03:58:22.854477 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.854399    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 03:58:22.855430 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.855411    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 03:58:22.855777 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.855416    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tc4d\""
Sep 09 03:58:22.855777 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.855526    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 03:58:22.855925 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.855555    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 03:58:22.856157 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.856137    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5sc28"
Sep 09 03:58:22.856547 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.856512    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:58:22.856788 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.856767    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h7965\""
Sep 09 03:58:22.856990 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.856968    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 03:58:22.858407 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.858390    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:22.858657 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.858442    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 03:58:22.859151 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.859028    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 03:58:22.860904 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.859265    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vtkl7\""
Sep 09 03:58:22.860904 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.859653    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 03:58:22.860904 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.860788    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-26d7m\""
Sep 09 03:58:22.861639 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.861620    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:22.861733 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:22.861680    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:22.863306 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.863288    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 03:58:22.863560 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.863543    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 03:58:22.863670 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.863650    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 03:58:22.869912 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.869887    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:22.874613 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.872675    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 03:58:22.874613 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.872681    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x8zjl\""
Sep 09 03:58:22.874613 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.873763    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.874840 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.874631    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 03:58:22.875841 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.875809    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 03:58:22.877033 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.877009    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dkvrf\""
Sep 09 03:58:22.877299 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.877279    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 03:58:22.878753 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.878722    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 03:58:22.879098 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.879076    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xcr59"
Sep 09 03:58:22.879421 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.879403    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 03:58:22.891665 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.891643    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xcr59"
Sep 09 03:58:22.910978 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.910957    2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:58:22.940900 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.940808    2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 03:58:22.948970 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.948943    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/18092042-a374-4054-9dfa-6b0a511d7c0f-hosts-file\") pod \"node-resolver-86q49\" (UID: \"18092042-a374-4054-9dfa-6b0a511d7c0f\") " pod="openshift-dns/node-resolver-86q49"
Sep 09 03:58:22.949087 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.948983    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67c61063-131e-4cc1-84ad-889cded7b195-host\") pod \"node-ca-k29sf\" (UID: \"67c61063-131e-4cc1-84ad-889cded7b195\") " pod="openshift-image-registry/node-ca-k29sf"
Sep 09 03:58:22.949087 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949012    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-run-netns\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.949087 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949036    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0479cc6-6713-4586-afc7-86451c83d955-konnectivity-ca\") pod \"konnectivity-agent-5sc28\" (UID: \"d0479cc6-6713-4586-afc7-86451c83d955\") " pod="kube-system/konnectivity-agent-5sc28"
Sep 09 03:58:22.949087 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949061    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-var-lib-cni-bin\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.949363 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949087    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6fz\" (UniqueName: \"kubernetes.io/projected/9579406a-2f97-456a-8d1a-bb8db3d24ab6-kube-api-access-dd6fz\") pod \"network-metrics-daemon-4x7vw\" (UID: \"9579406a-2f97-456a-8d1a-bb8db3d24ab6\") " pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:22.949363 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949093    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/18092042-a374-4054-9dfa-6b0a511d7c0f-hosts-file\") pod \"node-resolver-86q49\" (UID: \"18092042-a374-4054-9dfa-6b0a511d7c0f\") " pod="openshift-dns/node-resolver-86q49"
Sep 09 03:58:22.949363 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949110    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6slnl\" (UniqueName: \"kubernetes.io/projected/14b88a87-e2b4-41c5-97d2-b429d360db45-kube-api-access-6slnl\") pod \"iptables-alerter-xf8kb\" (UID: \"14b88a87-e2b4-41c5-97d2-b429d360db45\") " pod="openshift-network-operator/iptables-alerter-xf8kb"
Sep 09 03:58:22.949363 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949135    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-kubelet\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.949363 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949163    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-sysconfig\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.949363 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949174    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67c61063-131e-4cc1-84ad-889cded7b195-host\") pod \"node-ca-k29sf\" (UID: \"67c61063-131e-4cc1-84ad-889cded7b195\") " pod="openshift-image-registry/node-ca-k29sf"
Sep 09 03:58:22.949363 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949188    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-sysctl-conf\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.949363 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949228    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5b1820e-2357-48d0-ad83-f9859f396e8d-system-cni-dir\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:22.949363 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949261    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcx4k\" (UniqueName: \"kubernetes.io/projected/e85290a3-6d41-402c-b0bf-2a6f18e41c72-kube-api-access-qcx4k\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949383    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-run-ovn-kubernetes\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949413    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/746230ed-660b-47f9-9030-2004186a5844-ovn-node-metrics-cert\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949446    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-kubelet-config\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949489    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-cnibin\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949514    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949567    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-kubernetes\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949601    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5b1820e-2357-48d0-ad83-f9859f396e8d-cnibin\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949626    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-system-cni-dir\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949648    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-run-k8s-cni-cncf-io\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949670    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-run-netns\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949695    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-socket-dir\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949719    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5b1820e-2357-48d0-ad83-f9859f396e8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949743    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-var-lib-openvswitch\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949765    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-run-ovn\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949788    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-log-socket\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.949871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949813    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949836    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-modprobe-d\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949867    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kbcx\" (UniqueName: \"kubernetes.io/projected/b5b1820e-2357-48d0-ad83-f9859f396e8d-kube-api-access-6kbcx\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949939    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-multus-cni-dir\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949970    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs\") pod \"network-metrics-daemon-4x7vw\" (UID: \"9579406a-2f97-456a-8d1a-bb8db3d24ab6\") " pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.949995    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67c61063-131e-4cc1-84ad-889cded7b195-serviceca\") pod \"node-ca-k29sf\" (UID: \"67c61063-131e-4cc1-84ad-889cded7b195\") " pod="openshift-image-registry/node-ca-k29sf"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950020    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14b88a87-e2b4-41c5-97d2-b429d360db45-host-slash\") pod \"iptables-alerter-xf8kb\" (UID: \"14b88a87-e2b4-41c5-97d2-b429d360db45\") " pod="openshift-network-operator/iptables-alerter-xf8kb"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950046    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-run-openvswitch\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950074    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0479cc6-6713-4586-afc7-86451c83d955-agent-certs\") pod \"konnectivity-agent-5sc28\" (UID: \"d0479cc6-6713-4586-afc7-86451c83d955\") " pod="kube-system/konnectivity-agent-5sc28"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950099    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9e1143d-5214-469e-9340-c473864c59ce-etc-tuned\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950123    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5b1820e-2357-48d0-ad83-f9859f396e8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950150    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsr9n\" (UniqueName: \"kubernetes.io/projected/18092042-a374-4054-9dfa-6b0a511d7c0f-kube-api-access-tsr9n\") pod \"node-resolver-86q49\" (UID: \"18092042-a374-4054-9dfa-6b0a511d7c0f\") " pod="openshift-dns/node-resolver-86q49"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950178    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-registration-dir\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950203    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-systemd\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950233    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-lib-modules\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950258    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-hostroot\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.950808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950282    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-run-multus-certs\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950311    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950337    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4lt\" (UniqueName: \"kubernetes.io/projected/746230ed-660b-47f9-9030-2004186a5844-kube-api-access-mm4lt\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950364    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14b88a87-e2b4-41c5-97d2-b429d360db45-iptables-alerter-script\") pod \"iptables-alerter-xf8kb\" (UID: \"14b88a87-e2b4-41c5-97d2-b429d360db45\") " pod="openshift-network-operator/iptables-alerter-xf8kb"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950393    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-node-log\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950419    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/746230ed-660b-47f9-9030-2004186a5844-ovnkube-script-lib\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950445    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-sys-fs\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950470    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5b1820e-2357-48d0-ad83-f9859f396e8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950498    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e85290a3-6d41-402c-b0bf-2a6f18e41c72-multus-daemon-config\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950557    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-slash\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950586    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-device-dir\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950610    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-sys\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950639    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-var-lib-kubelet\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950671    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18092042-a374-4054-9dfa-6b0a511d7c0f-tmp-dir\") pod \"node-resolver-86q49\" (UID: \"18092042-a374-4054-9dfa-6b0a511d7c0f\") " pod="openshift-dns/node-resolver-86q49"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950710    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8pfl\" (UniqueName: \"kubernetes.io/projected/67c61063-131e-4cc1-84ad-889cded7b195-kube-api-access-m8pfl\") pod \"node-ca-k29sf\" (UID: \"67c61063-131e-4cc1-84ad-889cded7b195\") " pod="openshift-image-registry/node-ca-k29sf"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950738    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-systemd-units\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.951603 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950763    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-etc-openvswitch\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950787    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-cni-bin\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950880    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-cni-netd\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950914    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/746230ed-660b-47f9-9030-2004186a5844-ovnkube-config\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950942    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/746230ed-660b-47f9-9030-2004186a5844-env-overrides\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.950972    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-dbus\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951021    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfpcx\" (UniqueName: \"kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx\") pod \"network-check-target-njsng\" (UID: \"ccb3f1f6-8987-454c-8505-4adc83697891\") " pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951049    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-run\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951078    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-var-lib-kubelet\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951104    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-host\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951130    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5b1820e-2357-48d0-ad83-f9859f396e8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951158    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e85290a3-6d41-402c-b0bf-2a6f18e41c72-cni-binary-copy\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951183    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-multus-socket-dir-parent\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951197    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18092042-a374-4054-9dfa-6b0a511d7c0f-tmp-dir\") pod \"node-resolver-86q49\" (UID: \"18092042-a374-4054-9dfa-6b0a511d7c0f\") " pod="openshift-dns/node-resolver-86q49"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951211    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kw9g\" (UniqueName: \"kubernetes.io/projected/8f0d0434-5124-43eb-87a8-742b8a924d1b-kube-api-access-2kw9g\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951278    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-sysctl-d\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951302    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9e1143d-5214-469e-9340-c473864c59ce-tmp\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.952413 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951326    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-etc-kubernetes\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.953280 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951348    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-run-systemd\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:22.953280 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951371    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:22.953280 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951393    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxfsw\" (UniqueName: \"kubernetes.io/projected/b9e1143d-5214-469e-9340-c473864c59ce-kube-api-access-kxfsw\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:22.953280 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951416    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-var-lib-cni-multus\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.953280 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951460    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-multus-conf-dir\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:22.953280 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:22.951606    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:22.953280 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:22.951879    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs podName:9579406a-2f97-456a-8d1a-bb8db3d24ab6 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:23.451840715 +0000 UTC m=+2.978211406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs") pod "network-metrics-daemon-4x7vw" (UID: "9579406a-2f97-456a-8d1a-bb8db3d24ab6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:22.953280 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.951906    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67c61063-131e-4cc1-84ad-889cded7b195-serviceca\") pod \"node-ca-k29sf\" (UID: \"67c61063-131e-4cc1-84ad-889cded7b195\") " pod="openshift-image-registry/node-ca-k29sf"
Sep 09 03:58:22.956885 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.956860    2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 03:58:22.962200 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.961757    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6fz\" (UniqueName: \"kubernetes.io/projected/9579406a-2f97-456a-8d1a-bb8db3d24ab6-kube-api-access-dd6fz\") pod \"network-metrics-daemon-4x7vw\" (UID: \"9579406a-2f97-456a-8d1a-bb8db3d24ab6\") " pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:22.962200 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.962163    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8pfl\" (UniqueName: \"kubernetes.io/projected/67c61063-131e-4cc1-84ad-889cded7b195-kube-api-access-m8pfl\") pod \"node-ca-k29sf\" (UID: \"67c61063-131e-4cc1-84ad-889cded7b195\") " pod="openshift-image-registry/node-ca-k29sf"
Sep 09 03:58:22.962624 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.962598    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsr9n\" (UniqueName: \"kubernetes.io/projected/18092042-a374-4054-9dfa-6b0a511d7c0f-kube-api-access-tsr9n\") pod \"node-resolver-86q49\" (UID: \"18092042-a374-4054-9dfa-6b0a511d7c0f\") " pod="openshift-dns/node-resolver-86q49"
Sep 09 03:58:22.987740 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.987690    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal" event={"ID":"3b2974e26bc83af5f46cffe86e53f1fa","Type":"ContainerStarted","Data":"889f2a4162528c098da6e6c45e543607469bc1e6d0e2ea309535ab59fc1bc0dc"}
Sep 09 03:58:22.989236 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:22.989205    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-219.ec2.internal" event={"ID":"5cbec884f0899e64243f4816842f939e","Type":"ContainerStarted","Data":"c9cb9c3a77a37783f68fed0fa7f708723a440934477a38f15195b51e381e7c7f"}
Sep 09 03:58:23.051779 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.051746    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9e1143d-5214-469e-9340-c473864c59ce-tmp\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.051930 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.051794    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-etc-kubernetes\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.051930 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.051820    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-run-systemd\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.051930 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.051848    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.051930 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.051872    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxfsw\" (UniqueName: \"kubernetes.io/projected/b9e1143d-5214-469e-9340-c473864c59ce-kube-api-access-kxfsw\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.051930 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.051896    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-var-lib-cni-multus\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.051930 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.051920    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-multus-conf-dir\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.051945    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-run-netns\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.051969    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0479cc6-6713-4586-afc7-86451c83d955-konnectivity-ca\") pod \"konnectivity-agent-5sc28\" (UID: \"d0479cc6-6713-4586-afc7-86451c83d955\") " pod="kube-system/konnectivity-agent-5sc28"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.051991    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-var-lib-cni-bin\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052017    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6slnl\" (UniqueName: \"kubernetes.io/projected/14b88a87-e2b4-41c5-97d2-b429d360db45-kube-api-access-6slnl\") pod \"iptables-alerter-xf8kb\" (UID: \"14b88a87-e2b4-41c5-97d2-b429d360db45\") " pod="openshift-network-operator/iptables-alerter-xf8kb"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052040    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-kubelet\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052064    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-sysconfig\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052084    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-sysctl-conf\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052108    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5b1820e-2357-48d0-ad83-f9859f396e8d-system-cni-dir\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052130    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcx4k\" (UniqueName: \"kubernetes.io/projected/e85290a3-6d41-402c-b0bf-2a6f18e41c72-kube-api-access-qcx4k\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052141    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-multus-conf-dir\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052153    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-run-ovn-kubernetes\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052190    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-run-ovn-kubernetes\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052190    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/746230ed-660b-47f9-9030-2004186a5844-ovn-node-metrics-cert\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052227    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-kubelet-config\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:23.052253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052251    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-cnibin\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052273    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052297    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-kubernetes\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052319    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5b1820e-2357-48d0-ad83-f9859f396e8d-cnibin\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052341    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-system-cni-dir\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052365    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-run-k8s-cni-cncf-io\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052399    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-run-netns\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052431    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-socket-dir\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052456    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5b1820e-2357-48d0-ad83-f9859f396e8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052484    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-var-lib-openvswitch\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052506    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-run-ovn\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052547    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-log-socket\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052571    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052597    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-modprobe-d\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052602    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-cnibin\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052621    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kbcx\" (UniqueName: \"kubernetes.io/projected/b5b1820e-2357-48d0-ad83-f9859f396e8d-kube-api-access-6kbcx\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052646    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-multus-cni-dir\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.052894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052652    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-run-netns\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052705    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-multus-cni-dir\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052749    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14b88a87-e2b4-41c5-97d2-b429d360db45-host-slash\") pod \"iptables-alerter-xf8kb\" (UID: \"14b88a87-e2b4-41c5-97d2-b429d360db45\") " pod="openshift-network-operator/iptables-alerter-xf8kb"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052776    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-run-openvswitch\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052810    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0479cc6-6713-4586-afc7-86451c83d955-agent-certs\") pod \"konnectivity-agent-5sc28\" (UID: \"d0479cc6-6713-4586-afc7-86451c83d955\") " pod="kube-system/konnectivity-agent-5sc28"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052836    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9e1143d-5214-469e-9340-c473864c59ce-etc-tuned\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052859    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5b1820e-2357-48d0-ad83-f9859f396e8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052888    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-registration-dir\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052913    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-systemd\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052937    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-lib-modules\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052961    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-hostroot\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.052992    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-run-multus-certs\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053022    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053069    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4lt\" (UniqueName: \"kubernetes.io/projected/746230ed-660b-47f9-9030-2004186a5844-kube-api-access-mm4lt\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053097    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14b88a87-e2b4-41c5-97d2-b429d360db45-iptables-alerter-script\") pod \"iptables-alerter-xf8kb\" (UID: \"14b88a87-e2b4-41c5-97d2-b429d360db45\") " pod="openshift-network-operator/iptables-alerter-xf8kb"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053123    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-node-log\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053150    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/746230ed-660b-47f9-9030-2004186a5844-ovnkube-script-lib\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.053706 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053176    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-sys-fs\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053187    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0479cc6-6713-4586-afc7-86451c83d955-konnectivity-ca\") pod \"konnectivity-agent-5sc28\" (UID: \"d0479cc6-6713-4586-afc7-86451c83d955\") " pod="kube-system/konnectivity-agent-5sc28"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053203    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5b1820e-2357-48d0-ad83-f9859f396e8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053233    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e85290a3-6d41-402c-b0bf-2a6f18e41c72-multus-daemon-config\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053243    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-var-lib-cni-bin\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053262    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-slash\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053289    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-device-dir\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053305    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-kubelet\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053316    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-sys\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053342    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-var-lib-kubelet\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053351    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-sysconfig\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053371    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-systemd-units\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053395    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-etc-openvswitch\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053415    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-cni-bin\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053436    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-cni-netd\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053460    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/746230ed-660b-47f9-9030-2004186a5844-ovnkube-config\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053471    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-sysctl-conf\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.054244 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053489    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/746230ed-660b-47f9-9030-2004186a5844-env-overrides\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053510    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5b1820e-2357-48d0-ad83-f9859f396e8d-system-cni-dir\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053541    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-dbus\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053570    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfpcx\" (UniqueName: \"kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx\") pod \"network-check-target-njsng\" (UID: \"ccb3f1f6-8987-454c-8505-4adc83697891\") " pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053595    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-run\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053631    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-var-lib-kubelet\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053659    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-host\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053686    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5b1820e-2357-48d0-ad83-f9859f396e8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053711    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e85290a3-6d41-402c-b0bf-2a6f18e41c72-cni-binary-copy\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053739    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-multus-socket-dir-parent\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053791    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-kubelet-config\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053779    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kw9g\" (UniqueName: \"kubernetes.io/projected/8f0d0434-5124-43eb-87a8-742b8a924d1b-kube-api-access-2kw9g\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053841    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-sysctl-d\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053845    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.053960    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-kubernetes\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054001    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5b1820e-2357-48d0-ad83-f9859f396e8d-cnibin\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054047    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-system-cni-dir\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.055081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054085    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-run-k8s-cni-cncf-io\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054118    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-run-netns\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054157    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-sysctl-d\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054205    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054229    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-socket-dir\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054244    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-etc-kubernetes\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054282    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-var-lib-openvswitch\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054286    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-log-socket\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054318    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-run-ovn\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054355    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-run-systemd\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.054357    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.054413    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret podName:a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db nodeName:}" failed. No retries permitted until 2026-09-09 03:58:23.554395183 +0000 UTC m=+3.080765890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret") pod "global-pull-secret-syncer-kpwhr" (UID: "a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054422    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-modprobe-d\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054596    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14b88a87-e2b4-41c5-97d2-b429d360db45-host-slash\") pod \"iptables-alerter-xf8kb\" (UID: \"14b88a87-e2b4-41c5-97d2-b429d360db45\") " pod="openshift-network-operator/iptables-alerter-xf8kb"
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.054643    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-run-openvswitch\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.055984 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.055254    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5b1820e-2357-48d0-ad83-f9859f396e8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.057154 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.057129    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-device-dir\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.057483 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.057458    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-var-lib-cni-multus\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.058029 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.058002    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0479cc6-6713-4586-afc7-86451c83d955-agent-certs\") pod \"konnectivity-agent-5sc28\" (UID: \"d0479cc6-6713-4586-afc7-86451c83d955\") " pod="kube-system/konnectivity-agent-5sc28"
Sep 09 03:58:23.058166 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.058147    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-sys\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.058272 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.058172    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5b1820e-2357-48d0-ad83-f9859f396e8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.058272 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.058213    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-var-lib-kubelet\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.058377 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.058288    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-systemd-units\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.058377 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.058290    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-etc-systemd\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.058377 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.058337    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-etc-openvswitch\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.058557 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.058386    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-lib-modules\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.058557 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.058386    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-registration-dir\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.058557 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.058425    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-cni-bin\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.058557 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.058439    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-hostroot\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.058557 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.058490    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.058821 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.058561    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f0d0434-5124-43eb-87a8-742b8a924d1b-sys-fs\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.059029 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.059007    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/746230ed-660b-47f9-9030-2004186a5844-ovnkube-config\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.059115 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.059072    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14b88a87-e2b4-41c5-97d2-b429d360db45-iptables-alerter-script\") pod \"iptables-alerter-xf8kb\" (UID: \"14b88a87-e2b4-41c5-97d2-b429d360db45\") " pod="openshift-network-operator/iptables-alerter-xf8kb"
Sep 09 03:58:23.059174 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.059075    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-host-run-multus-certs\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.059237 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.059190    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-node-log\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.059452 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.059416    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/746230ed-660b-47f9-9030-2004186a5844-env-overrides\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.059544 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.059512    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-dbus\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:23.059694 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.059676    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/746230ed-660b-47f9-9030-2004186a5844-ovnkube-script-lib\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.060294 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.060266    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5b1820e-2357-48d0-ad83-f9859f396e8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.060383 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.060351    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-run\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.060383 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.060366    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e85290a3-6d41-402c-b0bf-2a6f18e41c72-multus-daemon-config\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.060488 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.060392    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-var-lib-kubelet\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.060488 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.060427    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9e1143d-5214-469e-9340-c473864c59ce-host\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.060488 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.060463    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e85290a3-6d41-402c-b0bf-2a6f18e41c72-multus-socket-dir-parent\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.060488 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.060487    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-cni-netd\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.060783 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.060545    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5b1820e-2357-48d0-ad83-f9859f396e8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.060783 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.060573    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/746230ed-660b-47f9-9030-2004186a5844-host-slash\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.061004 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.060984    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/746230ed-660b-47f9-9030-2004186a5844-ovn-node-metrics-cert\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.061747 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.061725    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9e1143d-5214-469e-9340-c473864c59ce-tmp\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.062549 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.062478    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e85290a3-6d41-402c-b0bf-2a6f18e41c72-cni-binary-copy\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.064328 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.064207    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcx4k\" (UniqueName: \"kubernetes.io/projected/e85290a3-6d41-402c-b0bf-2a6f18e41c72-kube-api-access-qcx4k\") pod \"multus-z5ftt\" (UID: \"e85290a3-6d41-402c-b0bf-2a6f18e41c72\") " pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.064328 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.064319    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9e1143d-5214-469e-9340-c473864c59ce-etc-tuned\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.065554 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.065533    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:58:23.065651 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.065564    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:58:23.065651 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.065590    2553 projected.go:196] Error preparing data for projected volume kube-api-access-pfpcx for pod openshift-network-diagnostics/network-check-target-njsng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:23.065651 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.065647    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx podName:ccb3f1f6-8987-454c-8505-4adc83697891 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:23.565628881 +0000 UTC m=+3.091999574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pfpcx" (UniqueName: "kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx") pod "network-check-target-njsng" (UID: "ccb3f1f6-8987-454c-8505-4adc83697891") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:23.067553 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.067457    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kw9g\" (UniqueName: \"kubernetes.io/projected/8f0d0434-5124-43eb-87a8-742b8a924d1b-kube-api-access-2kw9g\") pod \"aws-ebs-csi-driver-node-6g26b\" (UID: \"8f0d0434-5124-43eb-87a8-742b8a924d1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.067553 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.067461    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4lt\" (UniqueName: \"kubernetes.io/projected/746230ed-660b-47f9-9030-2004186a5844-kube-api-access-mm4lt\") pod \"ovnkube-node-v45h8\" (UID: \"746230ed-660b-47f9-9030-2004186a5844\") " pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.067709 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.067662    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxfsw\" (UniqueName: \"kubernetes.io/projected/b9e1143d-5214-469e-9340-c473864c59ce-kube-api-access-kxfsw\") pod \"tuned-jdmwv\" (UID: \"b9e1143d-5214-469e-9340-c473864c59ce\") " pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.068033 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.067996    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kbcx\" (UniqueName: \"kubernetes.io/projected/b5b1820e-2357-48d0-ad83-f9859f396e8d-kube-api-access-6kbcx\") pod \"multus-additional-cni-plugins-pj8mr\" (UID: \"b5b1820e-2357-48d0-ad83-f9859f396e8d\") " pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.069863 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.069839    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6slnl\" (UniqueName: \"kubernetes.io/projected/14b88a87-e2b4-41c5-97d2-b429d360db45-kube-api-access-6slnl\") pod \"iptables-alerter-xf8kb\" (UID: \"14b88a87-e2b4-41c5-97d2-b429d360db45\") " pod="openshift-network-operator/iptables-alerter-xf8kb"
Sep 09 03:58:23.151730 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.151640    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k29sf"
Sep 09 03:58:23.162883 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.162858    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-86q49"
Sep 09 03:58:23.182871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.182713    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xf8kb"
Sep 09 03:58:23.190723 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.190702    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:23.198763 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.198742    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jdmwv"
Sep 09 03:58:23.208191 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.208171    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5sc28"
Sep 09 03:58:23.218165 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.218012    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b"
Sep 09 03:58:23.225932 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.225914    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pj8mr"
Sep 09 03:58:23.233812 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.233784    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5ftt"
Sep 09 03:58:23.457085 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.457001    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs\") pod \"network-metrics-daemon-4x7vw\" (UID: \"9579406a-2f97-456a-8d1a-bb8db3d24ab6\") " pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:23.457228 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.457149    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:23.457228 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.457220    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs podName:9579406a-2f97-456a-8d1a-bb8db3d24ab6 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:24.457200585 +0000 UTC m=+3.983571278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs") pod "network-metrics-daemon-4x7vw" (UID: "9579406a-2f97-456a-8d1a-bb8db3d24ab6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:23.557369 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.557331    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:23.557557 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.557481    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:23.557606 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.557563    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret podName:a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db nodeName:}" failed. No retries permitted until 2026-09-09 03:58:24.557541862 +0000 UTC m=+4.083912561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret") pod "global-pull-secret-syncer-kpwhr" (UID: "a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:23.658120 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.658084    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfpcx\" (UniqueName: \"kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx\") pod \"network-check-target-njsng\" (UID: \"ccb3f1f6-8987-454c-8505-4adc83697891\") " pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:23.658294 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.658275    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:58:23.658340 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.658302    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:58:23.658340 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.658316    2553 projected.go:196] Error preparing data for projected volume kube-api-access-pfpcx for pod openshift-network-diagnostics/network-check-target-njsng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:23.658411 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.658398    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx podName:ccb3f1f6-8987-454c-8505-4adc83697891 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:24.658376094 +0000 UTC m=+4.184746787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfpcx" (UniqueName: "kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx") pod "network-check-target-njsng" (UID: "ccb3f1f6-8987-454c-8505-4adc83697891") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:23.896323 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.896245    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 03:53:23 +0000 UTC" deadline="2028-06-08 01:58:28.437714262 +0000 UTC"
Sep 09 03:58:23.896323 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.896278    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15310h0m4.541442896s"
Sep 09 03:58:23.983076 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:23.983049    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:23.983248 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:23.983187    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:24.110369 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:24.110336    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0479cc6_6713_4586_afc7_86451c83d955.slice/crio-eac7eaba3fbf0183c0a4e230908440668dd788c0c4a0086f965795c9b51b5a4b WatchSource:0}: Error finding container eac7eaba3fbf0183c0a4e230908440668dd788c0c4a0086f965795c9b51b5a4b: Status 404 returned error can't find the container with id eac7eaba3fbf0183c0a4e230908440668dd788c0c4a0086f965795c9b51b5a4b
Sep 09 03:58:24.111011 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:24.110991    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746230ed_660b_47f9_9030_2004186a5844.slice/crio-3b706d7632032f019196cea4adc97ab9903c9f28331298f839fb07195c75f947 WatchSource:0}: Error finding container 3b706d7632032f019196cea4adc97ab9903c9f28331298f839fb07195c75f947: Status 404 returned error can't find the container with id 3b706d7632032f019196cea4adc97ab9903c9f28331298f839fb07195c75f947
Sep 09 03:58:24.132913 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:24.132859    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b88a87_e2b4_41c5_97d2_b429d360db45.slice/crio-4a5d40ea005b00b1bd5fba18e75aa63325e600b1027fbf2e6ce2bd459e6e7cce WatchSource:0}: Error finding container 4a5d40ea005b00b1bd5fba18e75aa63325e600b1027fbf2e6ce2bd459e6e7cce: Status 404 returned error can't find the container with id 4a5d40ea005b00b1bd5fba18e75aa63325e600b1027fbf2e6ce2bd459e6e7cce
Sep 09 03:58:24.133783 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:24.133748    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85290a3_6d41_402c_b0bf_2a6f18e41c72.slice/crio-aa8a8af644c168d03ca9713dcbbb32008e8b20ffa2f05791cd8ce63919144949 WatchSource:0}: Error finding container aa8a8af644c168d03ca9713dcbbb32008e8b20ffa2f05791cd8ce63919144949: Status 404 returned error can't find the container with id aa8a8af644c168d03ca9713dcbbb32008e8b20ffa2f05791cd8ce63919144949
Sep 09 03:58:24.136691 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:24.136636    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18092042_a374_4054_9dfa_6b0a511d7c0f.slice/crio-429a91a4a4e6676e9c6517814e693e64a772851e498fad7738d3db713551d6ea WatchSource:0}: Error finding container 429a91a4a4e6676e9c6517814e693e64a772851e498fad7738d3db713551d6ea: Status 404 returned error can't find the container with id 429a91a4a4e6676e9c6517814e693e64a772851e498fad7738d3db713551d6ea
Sep 09 03:58:24.142384 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:24.142358    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0d0434_5124_43eb_87a8_742b8a924d1b.slice/crio-a17474375881d4a9972632ee538b6a50a552ca7d8f79a223d34bc7ee7e758601 WatchSource:0}: Error finding container a17474375881d4a9972632ee538b6a50a552ca7d8f79a223d34bc7ee7e758601: Status 404 returned error can't find the container with id a17474375881d4a9972632ee538b6a50a552ca7d8f79a223d34bc7ee7e758601
Sep 09 03:58:24.143194 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:24.143165    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e1143d_5214_469e_9340_c473864c59ce.slice/crio-685618f9c28d420678e6227c9e8aa988aed7b96954637a775265d269006d5819 WatchSource:0}: Error finding container 685618f9c28d420678e6227c9e8aa988aed7b96954637a775265d269006d5819: Status 404 returned error can't find the container with id 685618f9c28d420678e6227c9e8aa988aed7b96954637a775265d269006d5819
Sep 09 03:58:24.144880 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:24.144793    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c61063_131e_4cc1_84ad_889cded7b195.slice/crio-81def001ba32cfc349bb7ea2637633f04c12cb2a2b3d872ba44edeb613a1a49d WatchSource:0}: Error finding container 81def001ba32cfc349bb7ea2637633f04c12cb2a2b3d872ba44edeb613a1a49d: Status 404 returned error can't find the container with id 81def001ba32cfc349bb7ea2637633f04c12cb2a2b3d872ba44edeb613a1a49d
Sep 09 03:58:24.146099 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:58:24.145993    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b1820e_2357_48d0_ad83_f9859f396e8d.slice/crio-c54844b9905cdc47ea86afdc0d115c78369000c8b6c7acc1a3ddff9084b13b66 WatchSource:0}: Error finding container c54844b9905cdc47ea86afdc0d115c78369000c8b6c7acc1a3ddff9084b13b66: Status 404 returned error can't find the container with id c54844b9905cdc47ea86afdc0d115c78369000c8b6c7acc1a3ddff9084b13b66
Sep 09 03:58:24.464942 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:24.464742    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs\") pod \"network-metrics-daemon-4x7vw\" (UID: \"9579406a-2f97-456a-8d1a-bb8db3d24ab6\") " pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:24.465055 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:24.464909    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:24.465108 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:24.465059    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs podName:9579406a-2f97-456a-8d1a-bb8db3d24ab6 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:26.465039821 +0000 UTC m=+5.991410513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs") pod "network-metrics-daemon-4x7vw" (UID: "9579406a-2f97-456a-8d1a-bb8db3d24ab6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:24.566298 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:24.566260    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:24.566489 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:24.566418    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:24.566567 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:24.566511    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret podName:a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db nodeName:}" failed. No retries permitted until 2026-09-09 03:58:26.566488151 +0000 UTC m=+6.092858852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret") pod "global-pull-secret-syncer-kpwhr" (UID: "a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:24.667196 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:24.667103    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfpcx\" (UniqueName: \"kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx\") pod \"network-check-target-njsng\" (UID: \"ccb3f1f6-8987-454c-8505-4adc83697891\") " pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:24.667338 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:24.667279    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:58:24.667338 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:24.667297    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:58:24.667338 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:24.667309    2553 projected.go:196] Error preparing data for projected volume kube-api-access-pfpcx for pod openshift-network-diagnostics/network-check-target-njsng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:24.667507 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:24.667372    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx podName:ccb3f1f6-8987-454c-8505-4adc83697891 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:26.667351035 +0000 UTC m=+6.193721744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfpcx" (UniqueName: "kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx") pod "network-check-target-njsng" (UID: "ccb3f1f6-8987-454c-8505-4adc83697891") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:24.897307 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:24.897245    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 03:53:23 +0000 UTC" deadline="2028-04-07 21:11:51.501890107 +0000 UTC"
Sep 09 03:58:24.897307 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:24.897304    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13841h13m26.604590398s"
Sep 09 03:58:24.985065 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:24.985031    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:24.985211 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:24.985175    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:24.985644 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:24.985622    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:24.985748 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:24.985731    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:24.996703 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:24.996659    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pj8mr" event={"ID":"b5b1820e-2357-48d0-ad83-f9859f396e8d","Type":"ContainerStarted","Data":"c54844b9905cdc47ea86afdc0d115c78369000c8b6c7acc1a3ddff9084b13b66"}
Sep 09 03:58:25.002365 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:25.002316    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jdmwv" event={"ID":"b9e1143d-5214-469e-9340-c473864c59ce","Type":"ContainerStarted","Data":"685618f9c28d420678e6227c9e8aa988aed7b96954637a775265d269006d5819"}
Sep 09 03:58:25.015228 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:25.015197    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b" event={"ID":"8f0d0434-5124-43eb-87a8-742b8a924d1b","Type":"ContainerStarted","Data":"a17474375881d4a9972632ee538b6a50a552ca7d8f79a223d34bc7ee7e758601"}
Sep 09 03:58:25.036726 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:25.036580    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8" event={"ID":"746230ed-660b-47f9-9030-2004186a5844","Type":"ContainerStarted","Data":"3b706d7632032f019196cea4adc97ab9903c9f28331298f839fb07195c75f947"}
Sep 09 03:58:25.038410 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:25.038335    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5sc28" event={"ID":"d0479cc6-6713-4586-afc7-86451c83d955","Type":"ContainerStarted","Data":"eac7eaba3fbf0183c0a4e230908440668dd788c0c4a0086f965795c9b51b5a4b"}
Sep 09 03:58:25.046370 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:25.046146    2553 generic.go:356] "Generic (PLEG): container finished" podID="3b2974e26bc83af5f46cffe86e53f1fa" containerID="9ea311cbdd2ec6701e7d329884ba94fc6c82eb4bf9f27890193d6755e217bdea" exitCode=0
Sep 09 03:58:25.046370 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:25.046225    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal" event={"ID":"3b2974e26bc83af5f46cffe86e53f1fa","Type":"ContainerDied","Data":"9ea311cbdd2ec6701e7d329884ba94fc6c82eb4bf9f27890193d6755e217bdea"}
Sep 09 03:58:25.056740 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:25.056685    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-219.ec2.internal" event={"ID":"5cbec884f0899e64243f4816842f939e","Type":"ContainerStarted","Data":"ff75aaed57949c6891a70c7baff61db42ff2f7e305bf7a37350546d06d422cc8"}
Sep 09 03:58:25.085813 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:25.085762    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-219.ec2.internal" podStartSLOduration=2.085706342 podStartE2EDuration="2.085706342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:25.085321584 +0000 UTC m=+4.611692298" watchObservedRunningTime="2026-09-09 03:58:25.085706342 +0000 UTC m=+4.612077052"
Sep 09 03:58:25.089193 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:25.089164    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k29sf" event={"ID":"67c61063-131e-4cc1-84ad-889cded7b195","Type":"ContainerStarted","Data":"81def001ba32cfc349bb7ea2637633f04c12cb2a2b3d872ba44edeb613a1a49d"}
Sep 09 03:58:25.095979 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:25.095953    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-86q49" event={"ID":"18092042-a374-4054-9dfa-6b0a511d7c0f","Type":"ContainerStarted","Data":"429a91a4a4e6676e9c6517814e693e64a772851e498fad7738d3db713551d6ea"}
Sep 09 03:58:25.102439 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:25.102388    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5ftt" event={"ID":"e85290a3-6d41-402c-b0bf-2a6f18e41c72","Type":"ContainerStarted","Data":"aa8a8af644c168d03ca9713dcbbb32008e8b20ffa2f05791cd8ce63919144949"}
Sep 09 03:58:25.121753 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:25.121673    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xf8kb" event={"ID":"14b88a87-e2b4-41c5-97d2-b429d360db45","Type":"ContainerStarted","Data":"4a5d40ea005b00b1bd5fba18e75aa63325e600b1027fbf2e6ce2bd459e6e7cce"}
Sep 09 03:58:25.982273 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:25.982230    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:25.983015 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:25.982411    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:26.169702 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:26.169654    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal" event={"ID":"3b2974e26bc83af5f46cffe86e53f1fa","Type":"ContainerStarted","Data":"1fd8932566261ba74ecd88fd4d74e6bf9f01fb40bbfc5c3becddcf1629a1d050"}
Sep 09 03:58:26.189575 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:26.189502    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-219.ec2.internal" podStartSLOduration=3.1894780799999998 podStartE2EDuration="3.18947808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:26.18924133 +0000 UTC m=+5.715612043" watchObservedRunningTime="2026-09-09 03:58:26.18947808 +0000 UTC m=+5.715848831"
Sep 09 03:58:26.483591 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:26.482935    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs\") pod \"network-metrics-daemon-4x7vw\" (UID: \"9579406a-2f97-456a-8d1a-bb8db3d24ab6\") " pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:26.483591 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:26.483089    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:26.483591 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:26.483154    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs podName:9579406a-2f97-456a-8d1a-bb8db3d24ab6 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:30.483134098 +0000 UTC m=+10.009504792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs") pod "network-metrics-daemon-4x7vw" (UID: "9579406a-2f97-456a-8d1a-bb8db3d24ab6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:26.583432 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:26.583388    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:26.583638 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:26.583603    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:26.583707 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:26.583674    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret podName:a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db nodeName:}" failed. No retries permitted until 2026-09-09 03:58:30.583654054 +0000 UTC m=+10.110024774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret") pod "global-pull-secret-syncer-kpwhr" (UID: "a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:26.684787 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:26.684737    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfpcx\" (UniqueName: \"kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx\") pod \"network-check-target-njsng\" (UID: \"ccb3f1f6-8987-454c-8505-4adc83697891\") " pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:26.684998 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:26.684934    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:58:26.684998 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:26.684953    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:58:26.684998 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:26.684966    2553 projected.go:196] Error preparing data for projected volume kube-api-access-pfpcx for pod openshift-network-diagnostics/network-check-target-njsng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:26.685177 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:26.685029    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx podName:ccb3f1f6-8987-454c-8505-4adc83697891 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:30.685006678 +0000 UTC m=+10.211377372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfpcx" (UniqueName: "kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx") pod "network-check-target-njsng" (UID: "ccb3f1f6-8987-454c-8505-4adc83697891") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:26.991717 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:26.991682    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:26.992286 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:26.991869    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:26.992366 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:26.992339    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:26.992492 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:26.992467    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:27.983139 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:27.983107    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:27.983386 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:27.983252    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:28.985910 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:28.985380    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:28.985910 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:28.985547    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:28.985910 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:28.985581    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:28.985910 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:28.985723    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:29.982342 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:29.982270    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:29.982527 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:29.982425    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:30.522855 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:30.522818    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs\") pod \"network-metrics-daemon-4x7vw\" (UID: \"9579406a-2f97-456a-8d1a-bb8db3d24ab6\") " pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:30.523339 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:30.523031    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:30.523339 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:30.523113    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs podName:9579406a-2f97-456a-8d1a-bb8db3d24ab6 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:38.523090317 +0000 UTC m=+18.049461009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs") pod "network-metrics-daemon-4x7vw" (UID: "9579406a-2f97-456a-8d1a-bb8db3d24ab6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:30.623773 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:30.623739    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:30.623960 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:30.623923    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:30.624028 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:30.623982    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret podName:a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db nodeName:}" failed. No retries permitted until 2026-09-09 03:58:38.62396751 +0000 UTC m=+18.150338199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret") pod "global-pull-secret-syncer-kpwhr" (UID: "a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:30.724575 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:30.724535    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfpcx\" (UniqueName: \"kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx\") pod \"network-check-target-njsng\" (UID: \"ccb3f1f6-8987-454c-8505-4adc83697891\") " pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:30.724792 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:30.724771    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:58:30.724862 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:30.724804    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:58:30.724862 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:30.724819    2553 projected.go:196] Error preparing data for projected volume kube-api-access-pfpcx for pod openshift-network-diagnostics/network-check-target-njsng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:30.724982 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:30.724888    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx podName:ccb3f1f6-8987-454c-8505-4adc83697891 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:38.72486715 +0000 UTC m=+18.251237860 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfpcx" (UniqueName: "kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx") pod "network-check-target-njsng" (UID: "ccb3f1f6-8987-454c-8505-4adc83697891") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:30.984216 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:30.984134    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:30.984383 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:30.984262    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:30.984383 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:30.984287    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:30.984540 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:30.984411    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:31.982983 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:31.982406    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:31.982983 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:31.982563    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:32.982334 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:32.982298    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:32.982541 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:32.982438    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:32.982872 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:32.982850    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:32.983009 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:32.982980    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:33.982554 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:33.982524    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:33.982740 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:33.982640    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:34.983247 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:34.983214    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:34.983646 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:34.983214    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:34.983646 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:34.983351    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:34.983646 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:34.983402    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:35.983288 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:35.983256    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:35.983746 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:35.983385    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:36.982868 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:36.982832    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:36.983072 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:36.982882    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:36.983072 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:36.982999    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:36.983203 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:36.983101    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:37.982628 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:37.982595    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:37.983032 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:37.982704    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:38.579933 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:38.579893    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs\") pod \"network-metrics-daemon-4x7vw\" (UID: \"9579406a-2f97-456a-8d1a-bb8db3d24ab6\") " pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:38.580107 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:38.580031    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:38.580166 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:38.580108    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs podName:9579406a-2f97-456a-8d1a-bb8db3d24ab6 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:54.58008601 +0000 UTC m=+34.106456699 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs") pod "network-metrics-daemon-4x7vw" (UID: "9579406a-2f97-456a-8d1a-bb8db3d24ab6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:38.680852 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:38.680824    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:38.681000 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:38.680947    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:38.681000 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:38.680995    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret podName:a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db nodeName:}" failed. No retries permitted until 2026-09-09 03:58:54.680981488 +0000 UTC m=+34.207352177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret") pod "global-pull-secret-syncer-kpwhr" (UID: "a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:38.781981 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:38.781950    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfpcx\" (UniqueName: \"kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx\") pod \"network-check-target-njsng\" (UID: \"ccb3f1f6-8987-454c-8505-4adc83697891\") " pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:38.782114 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:38.782057    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:58:38.782114 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:38.782073    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:58:38.782114 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:38.782081    2553 projected.go:196] Error preparing data for projected volume kube-api-access-pfpcx for pod openshift-network-diagnostics/network-check-target-njsng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:38.782248 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:38.782124    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx podName:ccb3f1f6-8987-454c-8505-4adc83697891 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:54.78211189 +0000 UTC m=+34.308482579 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfpcx" (UniqueName: "kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx") pod "network-check-target-njsng" (UID: "ccb3f1f6-8987-454c-8505-4adc83697891") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:38.982569 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:38.982546    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:38.982720 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:38.982694    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:38.982977 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:38.982742    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:38.982977 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:38.982834    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:39.983045 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:39.982856    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:39.983489 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:39.983138    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:40.983094 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:40.983049    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:40.983528 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:40.983162    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:40.983528 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:40.983258    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:40.983528 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:40.983371    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:41.982362 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:41.982322    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:41.982535 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:41.982436    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:42.984300 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:42.984131    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:42.984300 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:42.984247    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:42.984918 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:42.984681    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:42.984918 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:42.984805    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:43.206720 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.206552    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8" event={"ID":"746230ed-660b-47f9-9030-2004186a5844","Type":"ContainerStarted","Data":"c91605099b5d926f9ca07a42073df50347a1f1e610e4040098422a5f41af4998"}
Sep 09 03:58:43.206872 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.206796    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8" event={"ID":"746230ed-660b-47f9-9030-2004186a5844","Type":"ContainerStarted","Data":"0b422df51e671f688d514bf31526870d8e012de368313a7b132a86ca60be56d7"}
Sep 09 03:58:43.206872 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.206840    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8" event={"ID":"746230ed-660b-47f9-9030-2004186a5844","Type":"ContainerStarted","Data":"bf3b1fe579fe626e9a7b221ad5894d0fc5c6411aea954dcdb15cc16e16ede550"}
Sep 09 03:58:43.206985 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.206886    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8" event={"ID":"746230ed-660b-47f9-9030-2004186a5844","Type":"ContainerStarted","Data":"fb74e1559fc22292f75d7ebab2a93a2b30e390f123aa99cd12fe787eeb917724"}
Sep 09 03:58:43.207963 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.207941    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5sc28" event={"ID":"d0479cc6-6713-4586-afc7-86451c83d955","Type":"ContainerStarted","Data":"787994e5829cdb1c265c36958a8875251d672b525ff20cc6a824a2fed872d456"}
Sep 09 03:58:43.209320 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.209296    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k29sf" event={"ID":"67c61063-131e-4cc1-84ad-889cded7b195","Type":"ContainerStarted","Data":"b8405df14918cee9ad8c9a0b569acedf28399d5a386607e67eb40024487f668b"}
Sep 09 03:58:43.210674 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.210653    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-86q49" event={"ID":"18092042-a374-4054-9dfa-6b0a511d7c0f","Type":"ContainerStarted","Data":"92c55dcb1e9b3a077b17184752c2578927e61113d2002fc1b203999559b279ee"}
Sep 09 03:58:43.212195 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.212175    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5ftt" event={"ID":"e85290a3-6d41-402c-b0bf-2a6f18e41c72","Type":"ContainerStarted","Data":"b72e025478baeabe279cd88e3d3b0757e544ff4b4d75cd68435b3653b95625bc"}
Sep 09 03:58:43.213672 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.213652    2553 generic.go:356] "Generic (PLEG): container finished" podID="b5b1820e-2357-48d0-ad83-f9859f396e8d" containerID="9b428853cd5b7b75960c76d7978f30200ec27f938e4e2f33caa2f35d0d0648f1" exitCode=0
Sep 09 03:58:43.213762 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.213712    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pj8mr" event={"ID":"b5b1820e-2357-48d0-ad83-f9859f396e8d","Type":"ContainerDied","Data":"9b428853cd5b7b75960c76d7978f30200ec27f938e4e2f33caa2f35d0d0648f1"}
Sep 09 03:58:43.217134 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.217104    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jdmwv" event={"ID":"b9e1143d-5214-469e-9340-c473864c59ce","Type":"ContainerStarted","Data":"2d29da47cf47d1f23484f97bb231a352e01e96d0057e7f66935cf5b85e6bf226"}
Sep 09 03:58:43.218538 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.218518    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b" event={"ID":"8f0d0434-5124-43eb-87a8-742b8a924d1b","Type":"ContainerStarted","Data":"8283b22f0f36336a5d5c3754b9a6adf34f250d5539119633956ec7fe2481558d"}
Sep 09 03:58:43.236775 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.236733    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5sc28" podStartSLOduration=3.6022214999999997 podStartE2EDuration="22.236721426s" totalImagesPullingTime="18.634499926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:43.222847363 +0000 UTC m=+22.749218062" watchObservedRunningTime="2026-09-09 03:58:43.236721426 +0000 UTC m=+22.763092116"
Sep 09 03:58:43.237023 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.236996    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k29sf" podStartSLOduration=3.617627049 podStartE2EDuration="22.236990791s" totalImagesPullingTime="18.619363742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:43.236815341 +0000 UTC m=+22.763186171" watchObservedRunningTime="2026-09-09 03:58:43.236990791 +0000 UTC m=+22.763361499"
Sep 09 03:58:43.282759 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.282686    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z5ftt" podStartSLOduration=3.57370227 podStartE2EDuration="22.282674888s" totalImagesPullingTime="18.708972618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:43.282264946 +0000 UTC m=+22.808635655" watchObservedRunningTime="2026-09-09 03:58:43.282674888 +0000 UTC m=+22.809045595"
Sep 09 03:58:43.297953 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.297915    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-86q49" podStartSLOduration=3.667530532 podStartE2EDuration="22.297900841s" totalImagesPullingTime="18.630370309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:43.297742428 +0000 UTC m=+22.824113137" watchObservedRunningTime="2026-09-09 03:58:43.297900841 +0000 UTC m=+22.824271551"
Sep 09 03:58:43.317898 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.317868    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jdmwv" podStartSLOduration=3.671208369 podStartE2EDuration="22.317857986s" totalImagesPullingTime="18.646649617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:43.317768394 +0000 UTC m=+22.844139104" watchObservedRunningTime="2026-09-09 03:58:43.317857986 +0000 UTC m=+22.844228696"
Sep 09 03:58:43.322711 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.322683    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:43.399253 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.399234    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:43.515139 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.515106    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:43.597792 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.597739    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:43.983079 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:43.983047    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:43.983252 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:43.983169    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:44.200184 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.200156    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:44.222354 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.222325    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xf8kb" event={"ID":"14b88a87-e2b4-41c5-97d2-b429d360db45","Type":"ContainerStarted","Data":"976de00ef526c8bc527332884220a378eb930df3a8ff9964f47ca49c4c7a9fdd"}
Sep 09 03:58:44.225888 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.225861    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8" event={"ID":"746230ed-660b-47f9-9030-2004186a5844","Type":"ContainerStarted","Data":"1508eb1220cea9aaeed14dbba8ade60e23ff1d2557fb118d34fbb1da7287f005"}
Sep 09 03:58:44.226007 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.225941    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8" event={"ID":"746230ed-660b-47f9-9030-2004186a5844","Type":"ContainerStarted","Data":"b97c15dca8ddd8162c7599cead0d26037201d64b2a2c102f9a77dbc7abaa3996"}
Sep 09 03:58:44.245066 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.244986    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xf8kb" podStartSLOduration=4.592023332 podStartE2EDuration="23.244972556s" totalImagesPullingTime="18.652949224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:44.240371733 +0000 UTC m=+23.766742441" watchObservedRunningTime="2026-09-09 03:58:44.244972556 +0000 UTC m=+23.771343265"
Sep 09 03:58:44.298220 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.298192    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5sc28"
Sep 09 03:58:44.298788 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.298769    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5sc28"
Sep 09 03:58:44.367265 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.367238    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:44.465968 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.465947    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:44.564694 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.564670    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:44.599135 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.599113    2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 03:58:44.665302 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.665266    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:44.908246 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.908117    2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T03:58:44.599131506Z","UUID":"9f41c3b4-59a7-4eac-a754-96566dcdf460","Handler":null,"Name":"","Endpoint":""}
Sep 09 03:58:44.909836 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.909815    2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 03:58:44.909967 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.909854    2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 03:58:44.982910 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.982859    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:44.983026 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:44.982993    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:44.983100 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:44.983081    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:44.983211 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:44.983189    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:45.075567 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:45.075539    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:45.134524 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:45.134497    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:45.230498 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:45.230421    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b" event={"ID":"8f0d0434-5124-43eb-87a8-742b8a924d1b","Type":"ContainerStarted","Data":"75797ece54b1acd01f7ea3be95273ada950a6b1df7ab96917618be23be4f92e4"}
Sep 09 03:58:45.230906 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:45.230789    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5sc28"
Sep 09 03:58:45.231303 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:45.231262    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5sc28"
Sep 09 03:58:45.299141 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:45.299102    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:45.399615 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:45.399588    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:45.540682 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:45.540654    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:45.561957 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:45.561939    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-86q49_18092042-a374-4054-9dfa-6b0a511d7c0f/dns-node-resolver/0.log"
Sep 09 03:58:45.634614 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:45.634417    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:45.793684 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:45.793592    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:45.893184 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:45.893154    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:45.982650 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:45.982623    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:45.982794 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:45.982740    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:46.235834 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:46.235301    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b" event={"ID":"8f0d0434-5124-43eb-87a8-742b8a924d1b","Type":"ContainerStarted","Data":"1b1fe219342549121084cc93ef267c91fd173f136782de3af998df2304ed68d9"}
Sep 09 03:58:46.239489 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:46.239462    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8" event={"ID":"746230ed-660b-47f9-9030-2004186a5844","Type":"ContainerStarted","Data":"2df6847be887945c7bb00f36a9da790f96168d8dd0363270e82c6af9ec5c2d53"}
Sep 09 03:58:46.561990 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:46.561912    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:46.666922 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:46.666886    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:46.763159 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:46.763128    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k29sf_67c61063-131e-4cc1-84ad-889cded7b195/node-ca/0.log"
Sep 09 03:58:46.982698 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:46.982631    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:46.982841 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:46.982738    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:46.982841 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:46.982800    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:46.982983 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:46.982960    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:47.983372 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:47.983190    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:47.983790 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:47.983456    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:48.225738 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:48.225668    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:48.247285 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:48.247259    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8" event={"ID":"746230ed-660b-47f9-9030-2004186a5844","Type":"ContainerStarted","Data":"a14858433e86cc2db92d6ac933135f2e36cd7dfe71917f351dfadc67e47c287e"}
Sep 09 03:58:48.247524 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:48.247506    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:48.249077 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:48.249056    2553 generic.go:356] "Generic (PLEG): container finished" podID="b5b1820e-2357-48d0-ad83-f9859f396e8d" containerID="b0a86513cf3303e7916eb61e4efc49ac4489657211bf5c0986234a465b3c2408" exitCode=0
Sep 09 03:58:48.249187 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:48.249086    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pj8mr" event={"ID":"b5b1820e-2357-48d0-ad83-f9859f396e8d","Type":"ContainerDied","Data":"b0a86513cf3303e7916eb61e4efc49ac4489657211bf5c0986234a465b3c2408"}
Sep 09 03:58:48.263077 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:48.263057    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:48.278597 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:48.278558    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g26b" podStartSLOduration=6.099510474 podStartE2EDuration="27.278546477s" totalImagesPullingTime="21.179036003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:46.259488006 +0000 UTC m=+25.785858717" watchObservedRunningTime="2026-09-09 03:58:48.278546477 +0000 UTC m=+27.804917227"
Sep 09 03:58:48.286146 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:48.286129    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:48.313839 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:48.313804    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8" podStartSLOduration=8.626748197 podStartE2EDuration="27.313792257s" totalImagesPullingTime="18.68704406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:48.279328493 +0000 UTC m=+27.805699201" watchObservedRunningTime="2026-09-09 03:58:48.313792257 +0000 UTC m=+27.840162968"
Sep 09 03:58:48.388819 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:48.388793    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:48.985274 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:48.985211    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:48.985914 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:48.985388    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:48.985914 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:48.985489    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:48.985914 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:48.985567    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:49.251922 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:49.251853    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:49.252050 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:49.251936    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:49.267078 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:49.267052    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:58:49.599935 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:49.599910    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kpwhr"]
Sep 09 03:58:49.600021 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:49.600009    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:49.600134 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:49.600116    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:49.602055 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:49.602034    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njsng"]
Sep 09 03:58:49.602157 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:49.602138    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:49.602249 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:49.602231    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:49.602831 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:49.602806    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4x7vw"]
Sep 09 03:58:49.602939 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:49.602878    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:49.603023 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:49.603002    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:50.255576 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:50.255548    2553 generic.go:356] "Generic (PLEG): container finished" podID="b5b1820e-2357-48d0-ad83-f9859f396e8d" containerID="b27bf4a41105490acf8920d88d0efac64bc1c4512f6505bbe8bc4f84a1aaad2f" exitCode=0
Sep 09 03:58:50.256232 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:50.255600    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pj8mr" event={"ID":"b5b1820e-2357-48d0-ad83-f9859f396e8d","Type":"ContainerDied","Data":"b27bf4a41105490acf8920d88d0efac64bc1c4512f6505bbe8bc4f84a1aaad2f"}
Sep 09 03:58:50.985922 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:50.985893    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:50.986081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:50.985980    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:50.986081 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:50.985986    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:50.986221 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:50.986092    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:50.986221 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:50.986148    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:50.986310 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:50.986232    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:51.290252 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:51.290228    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:51.724202 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:51.724016    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:51.902084 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:51.902059    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:52.052895 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:52.052798    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:52.154961 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:52.154932    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:52.259656 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:52.259629    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:52.261047 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:52.261027    2553 generic.go:356] "Generic (PLEG): container finished" podID="b5b1820e-2357-48d0-ad83-f9859f396e8d" containerID="a437a206c7980ae670fe21ae6db181f100b70b02e25b7ad5bc2804151ea7c257" exitCode=0
Sep 09 03:58:52.261156 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:52.261078    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pj8mr" event={"ID":"b5b1820e-2357-48d0-ad83-f9859f396e8d","Type":"ContainerDied","Data":"a437a206c7980ae670fe21ae6db181f100b70b02e25b7ad5bc2804151ea7c257"}
Sep 09 03:58:52.422694 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:52.422638    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:52.684889 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:52.684826    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:52.743539 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:52.743501    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:52.986563 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:52.986535    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:52.986725 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:52.986666    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:52.986725 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:52.986674    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:52.986829 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:52.986752    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:52.986829 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:52.986755    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:52.986921 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:52.986841    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:53.646517 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:53.646484    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:53.825618 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:53.825574    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:54.056350 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:54.056320    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:54.118224 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:54.118196    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:54.227638 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:54.227610    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:54.421371 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:54.421292    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:54.597547 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:54.597514    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs\") pod \"network-metrics-daemon-4x7vw\" (UID: \"9579406a-2f97-456a-8d1a-bb8db3d24ab6\") " pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:54.597683 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:54.597645    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:54.597743 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:54.597708    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs podName:9579406a-2f97-456a-8d1a-bb8db3d24ab6 nodeName:}" failed. No retries permitted until 2026-09-09 03:59:26.597693107 +0000 UTC m=+66.124063795 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs") pod "network-metrics-daemon-4x7vw" (UID: "9579406a-2f97-456a-8d1a-bb8db3d24ab6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:54.641307 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:54.641245    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:54.698350 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:54.698276    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:54.698729 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:54.698435    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:54.698729 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:54.698521    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret podName:a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db nodeName:}" failed. No retries permitted until 2026-09-09 03:59:26.698501529 +0000 UTC m=+66.224872220 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret") pod "global-pull-secret-syncer-kpwhr" (UID: "a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:54.799022 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:54.798987    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfpcx\" (UniqueName: \"kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx\") pod \"network-check-target-njsng\" (UID: \"ccb3f1f6-8987-454c-8505-4adc83697891\") " pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:54.799161 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:54.799123    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:58:54.799161 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:54.799151    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:58:54.799263 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:54.799165    2553 projected.go:196] Error preparing data for projected volume kube-api-access-pfpcx for pod openshift-network-diagnostics/network-check-target-njsng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:54.799263 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:54.799220    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx podName:ccb3f1f6-8987-454c-8505-4adc83697891 nodeName:}" failed. No retries permitted until 2026-09-09 03:59:26.79920274 +0000 UTC m=+66.325573427 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfpcx" (UniqueName: "kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx") pod "network-check-target-njsng" (UID: "ccb3f1f6-8987-454c-8505-4adc83697891") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:54.983139 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:54.983082    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:54.983139 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:54.983100    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:54.983139 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:54.983112    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:54.983413 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:54.983222    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:54.983635 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:54.983611    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:54.983724 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:54.983706    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:55.477975 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:55.477937    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:55.604568 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:55.604540    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:55.674636 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:55.674599    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:55.823893 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:55.823818    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:56.023723 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:56.023690    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:56.222635 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:56.222608    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:56.278206 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:56.278179    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:56.446688 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:56.446664    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:56.502824 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:56.502774    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:56.624411 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:56.624380    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:56.985734 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:56.985705    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:56.986118 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:56.985708    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:56.986118 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:56.985828    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:56.986118 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:56.985707    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:56.986118 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:56.985865    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:56.986118 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:56.985934    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:57.223436 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:57.223410    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:57.274594 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:57.274545    2553 generic.go:356] "Generic (PLEG): container finished" podID="b5b1820e-2357-48d0-ad83-f9859f396e8d" containerID="943e33b3007221b3ff5df5b7ce81d2b4acf8d68adedc886a2064a8881d13df4e" exitCode=0
Sep 09 03:58:57.274689 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:57.274599    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pj8mr" event={"ID":"b5b1820e-2357-48d0-ad83-f9859f396e8d","Type":"ContainerDied","Data":"943e33b3007221b3ff5df5b7ce81d2b4acf8d68adedc886a2064a8881d13df4e"}
Sep 09 03:58:57.626458 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:57.626385    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:58.020918 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:58.020891    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:58.206481 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:58.206455    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:58.266778 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:58.266756    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:58.279218 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:58.279165    2553 generic.go:356] "Generic (PLEG): container finished" podID="b5b1820e-2357-48d0-ad83-f9859f396e8d" containerID="cea2ec802f7d104da18eeb234b289773731ed5710adf2ad7beb3d9f4d7007dc9" exitCode=0
Sep 09 03:58:58.279308 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:58.279219    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pj8mr" event={"ID":"b5b1820e-2357-48d0-ad83-f9859f396e8d","Type":"ContainerDied","Data":"cea2ec802f7d104da18eeb234b289773731ed5710adf2ad7beb3d9f4d7007dc9"}
Sep 09 03:58:58.829000 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:58.828834    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:58.986305 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:58.986280    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:58:58.986305 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:58.986293    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:58:58.986519 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:58.986315    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:58:58.986519 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:58.986407    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:58:58.986519 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:58.986462    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:58:58.986614 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:58:58.986522    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:58:59.240120 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:59.240092    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:59.284749 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:59.284716    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pj8mr" event={"ID":"b5b1820e-2357-48d0-ad83-f9859f396e8d","Type":"ContainerStarted","Data":"8f9f483da0521573accec5c3dde7381d8d4dd8e39bdb857468b770fc1805dc31"}
Sep 09 03:58:59.319272 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:59.319230    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pj8mr" podStartSLOduration=8.968633497 podStartE2EDuration="38.319216101s" totalImagesPullingTime="29.350582604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:59.318821229 +0000 UTC m=+38.845191938" watchObservedRunningTime="2026-09-09 03:58:59.319216101 +0000 UTC m=+38.845586809"
Sep 09 03:58:59.639246 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:59.639217    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:59.829712 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:58:59.829687    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:00.421670 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:00.421638    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:00.524131 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:00.524099    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:00.983394 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:00.983365    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:59:00.983573 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:00.983463    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:59:00.983573 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:00.983530    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:59:00.983653 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:00.983641    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:59:00.983691 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:00.983683    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:59:00.983787 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:00.983769    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:59:01.451594 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:01.451515    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:01.510656 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:01.510632    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:02.030671 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:02.030640    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:02.240612 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:02.240583    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:02.432850 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:02.432786    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:02.982859 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:02.982830    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:59:02.982859 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:02.982858    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:59:02.983276 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:02.982914    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:59:02.983276 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:02.982831    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:59:02.983276 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:02.983019    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:59:02.983276 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:02.983106    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:59:04.982580 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:04.982545    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:59:04.982580 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:04.982578    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:59:04.983038 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:04.982545    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:59:04.983038 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:04.982642    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:59:04.983038 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:04.982735    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:59:04.983038 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:04.982803    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:59:06.982709 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:06.982682    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:59:06.983135 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:06.982713    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:59:06.983135 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:06.982682    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:59:06.983135 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:06.982788    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:59:06.983135 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:06.982860    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:59:06.983135 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:06.982976    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:59:08.221506 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:08.221459    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:08.316133 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:08.316107    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:08.420278 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:08.420248    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:08.516146 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:08.516090    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:08.982812 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:08.982788    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:59:08.982958 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:08.982816    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:59:08.982958 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:08.982793    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:59:08.982958 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:08.982868    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpwhr" podUID="a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db"
Sep 09 03:59:08.983078 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:08.982965    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njsng" podUID="ccb3f1f6-8987-454c-8505-4adc83697891"
Sep 09 03:59:08.983078 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:08.983030    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4x7vw" podUID="9579406a-2f97-456a-8d1a-bb8db3d24ab6"
Sep 09 03:59:09.404569 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:09.404476    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:09.494227 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:09.494196    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:09.594638 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:09.594597    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:09.695471 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:09.695390    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:09.796111 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:09.796085    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:10.683680 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.683613    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-219.ec2.internal" event="NodeReady"
Sep 09 03:59:10.684016 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.683702    2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 03:59:10.730852 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.730829    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lggsm"]
Sep 09 03:59:10.738038 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.738017    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8d765"]
Sep 09 03:59:10.738145 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.738130    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:10.740639 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.740622    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 03:59:10.740737 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.740672    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 03:59:10.740737 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.740622    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nrdh9\""
Sep 09 03:59:10.747270 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.747241    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lggsm"]
Sep 09 03:59:10.747383 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.747355    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:10.747715 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.747698    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8d765"]
Sep 09 03:59:10.750150 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.750130    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 03:59:10.750364 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.750345    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 03:59:10.750484 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.750392    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 03:59:10.750484 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.750392    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4kqm2\""
Sep 09 03:59:10.751171 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.751153    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 03:59:10.831740 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.831715    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kdvjc"]
Sep 09 03:59:10.843473 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.843431    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kdvjc"]
Sep 09 03:59:10.843574 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.843525    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kdvjc"
Sep 09 03:59:10.846033 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.846011    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 03:59:10.846033 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.846026    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 03:59:10.846273 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.846096    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2ccw8\""
Sep 09 03:59:10.846335 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.846315    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 03:59:10.885885 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.885863    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:10.922511 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.922487    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e8503b7-f81e-4115-aa5d-e38becb6e75a-crio-socket\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:10.922600 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.922516    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a976dc5-634b-425c-88f4-d8e2822e1911-metrics-tls\") pod \"dns-default-lggsm\" (UID: \"5a976dc5-634b-425c-88f4-d8e2822e1911\") " pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:10.922600 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.922538    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e8503b7-f81e-4115-aa5d-e38becb6e75a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:10.922600 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.922583    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qknj4\" (UniqueName: \"kubernetes.io/projected/5a976dc5-634b-425c-88f4-d8e2822e1911-kube-api-access-qknj4\") pod \"dns-default-lggsm\" (UID: \"5a976dc5-634b-425c-88f4-d8e2822e1911\") " pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:10.922696 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.922616    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a976dc5-634b-425c-88f4-d8e2822e1911-tmp-dir\") pod \"dns-default-lggsm\" (UID: \"5a976dc5-634b-425c-88f4-d8e2822e1911\") " pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:10.922696 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.922656    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-227m2\" (UniqueName: \"kubernetes.io/projected/5e8503b7-f81e-4115-aa5d-e38becb6e75a-kube-api-access-227m2\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:10.922696 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.922682    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e8503b7-f81e-4115-aa5d-e38becb6e75a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:10.922786 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.922704    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a976dc5-634b-425c-88f4-d8e2822e1911-config-volume\") pod \"dns-default-lggsm\" (UID: \"5a976dc5-634b-425c-88f4-d8e2822e1911\") " pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:10.922786 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.922760    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e8503b7-f81e-4115-aa5d-e38becb6e75a-data-volume\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:10.954046 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.953995    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:10.983427 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.983412    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:59:10.983622 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.983607    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:59:10.983782 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.983761    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:59:10.986034 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.986003    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 03:59:10.986124 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.986073    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 03:59:10.986124 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.986073    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 03:59:10.986124 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.986077    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-slrsq\""
Sep 09 03:59:10.986279 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.986114    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 03:59:10.986279 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:10.986214    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-krf5h\""
Sep 09 03:59:11.023190 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023170    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a976dc5-634b-425c-88f4-d8e2822e1911-config-volume\") pod \"dns-default-lggsm\" (UID: \"5a976dc5-634b-425c-88f4-d8e2822e1911\") " pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:11.023271 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023208    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e8503b7-f81e-4115-aa5d-e38becb6e75a-data-volume\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:11.023271 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023228    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e8503b7-f81e-4115-aa5d-e38becb6e75a-crio-socket\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:11.023271 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023242    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a976dc5-634b-425c-88f4-d8e2822e1911-metrics-tls\") pod \"dns-default-lggsm\" (UID: \"5a976dc5-634b-425c-88f4-d8e2822e1911\") " pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:11.023546 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023372    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e8503b7-f81e-4115-aa5d-e38becb6e75a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:11.023546 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023416    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfbd047c-3b5e-442f-938b-667a26e580f3-cert\") pod \"ingress-canary-kdvjc\" (UID: \"cfbd047c-3b5e-442f-938b-667a26e580f3\") " pod="openshift-ingress-canary/ingress-canary-kdvjc"
Sep 09 03:59:11.023546 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023432    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e8503b7-f81e-4115-aa5d-e38becb6e75a-crio-socket\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:11.023546 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023470    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qknj4\" (UniqueName: \"kubernetes.io/projected/5a976dc5-634b-425c-88f4-d8e2822e1911-kube-api-access-qknj4\") pod \"dns-default-lggsm\" (UID: \"5a976dc5-634b-425c-88f4-d8e2822e1911\") " pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:11.023546 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023512    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a976dc5-634b-425c-88f4-d8e2822e1911-tmp-dir\") pod \"dns-default-lggsm\" (UID: \"5a976dc5-634b-425c-88f4-d8e2822e1911\") " pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:11.023796 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023586    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227m2\" (UniqueName: \"kubernetes.io/projected/5e8503b7-f81e-4115-aa5d-e38becb6e75a-kube-api-access-227m2\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:11.023796 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023516    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e8503b7-f81e-4115-aa5d-e38becb6e75a-data-volume\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:11.023796 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023617    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e8503b7-f81e-4115-aa5d-e38becb6e75a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:11.023796 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023643    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rmmw\" (UniqueName: \"kubernetes.io/projected/cfbd047c-3b5e-442f-938b-667a26e580f3-kube-api-access-4rmmw\") pod \"ingress-canary-kdvjc\" (UID: \"cfbd047c-3b5e-442f-938b-667a26e580f3\") " pod="openshift-ingress-canary/ingress-canary-kdvjc"
Sep 09 03:59:11.023796 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023788    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a976dc5-634b-425c-88f4-d8e2822e1911-config-volume\") pod \"dns-default-lggsm\" (UID: \"5a976dc5-634b-425c-88f4-d8e2822e1911\") " pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:11.023958 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023801    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a976dc5-634b-425c-88f4-d8e2822e1911-tmp-dir\") pod \"dns-default-lggsm\" (UID: \"5a976dc5-634b-425c-88f4-d8e2822e1911\") " pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:11.023958 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.023919    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e8503b7-f81e-4115-aa5d-e38becb6e75a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:11.027297 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.027277    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a976dc5-634b-425c-88f4-d8e2822e1911-metrics-tls\") pod \"dns-default-lggsm\" (UID: \"5a976dc5-634b-425c-88f4-d8e2822e1911\") " pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:11.027367 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.027283    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e8503b7-f81e-4115-aa5d-e38becb6e75a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:11.031743 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.031723    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-227m2\" (UniqueName: \"kubernetes.io/projected/5e8503b7-f81e-4115-aa5d-e38becb6e75a-kube-api-access-227m2\") pod \"insights-runtime-extractor-8d765\" (UID: \"5e8503b7-f81e-4115-aa5d-e38becb6e75a\") " pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:11.032020 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.032003    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qknj4\" (UniqueName: \"kubernetes.io/projected/5a976dc5-634b-425c-88f4-d8e2822e1911-kube-api-access-qknj4\") pod \"dns-default-lggsm\" (UID: \"5a976dc5-634b-425c-88f4-d8e2822e1911\") " pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:11.048059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.048043    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:11.056204 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.056188    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8d765"
Sep 09 03:59:11.124508 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.124474    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfbd047c-3b5e-442f-938b-667a26e580f3-cert\") pod \"ingress-canary-kdvjc\" (UID: \"cfbd047c-3b5e-442f-938b-667a26e580f3\") " pod="openshift-ingress-canary/ingress-canary-kdvjc"
Sep 09 03:59:11.124693 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.124550    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rmmw\" (UniqueName: \"kubernetes.io/projected/cfbd047c-3b5e-442f-938b-667a26e580f3-kube-api-access-4rmmw\") pod \"ingress-canary-kdvjc\" (UID: \"cfbd047c-3b5e-442f-938b-667a26e580f3\") " pod="openshift-ingress-canary/ingress-canary-kdvjc"
Sep 09 03:59:11.128082 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.127426    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfbd047c-3b5e-442f-938b-667a26e580f3-cert\") pod \"ingress-canary-kdvjc\" (UID: \"cfbd047c-3b5e-442f-938b-667a26e580f3\") " pod="openshift-ingress-canary/ingress-canary-kdvjc"
Sep 09 03:59:11.134638 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.134577    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rmmw\" (UniqueName: \"kubernetes.io/projected/cfbd047c-3b5e-442f-938b-667a26e580f3-kube-api-access-4rmmw\") pod \"ingress-canary-kdvjc\" (UID: \"cfbd047c-3b5e-442f-938b-667a26e580f3\") " pod="openshift-ingress-canary/ingress-canary-kdvjc"
Sep 09 03:59:11.151830 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.151807    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kdvjc"
Sep 09 03:59:11.353298 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.353271    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lggsm"]
Sep 09 03:59:11.356160 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.355931    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8d765"]
Sep 09 03:59:11.356775 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:11.356754    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a976dc5_634b_425c_88f4_d8e2822e1911.slice/crio-8258c4f1628c091b77768f0a4ac45b1f87a61c1f0b6aeb36c0b6ba70e4d8118c WatchSource:0}: Error finding container 8258c4f1628c091b77768f0a4ac45b1f87a61c1f0b6aeb36c0b6ba70e4d8118c: Status 404 returned error can't find the container with id 8258c4f1628c091b77768f0a4ac45b1f87a61c1f0b6aeb36c0b6ba70e4d8118c
Sep 09 03:59:11.359521 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:11.359498    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8503b7_f81e_4115_aa5d_e38becb6e75a.slice/crio-92ca4033c0fc346d5672a4108b9cf8b59a616f0da33cdff88624c32ec1b1b5d4 WatchSource:0}: Error finding container 92ca4033c0fc346d5672a4108b9cf8b59a616f0da33cdff88624c32ec1b1b5d4: Status 404 returned error can't find the container with id 92ca4033c0fc346d5672a4108b9cf8b59a616f0da33cdff88624c32ec1b1b5d4
Sep 09 03:59:11.436373 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:11.436352    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbd047c_3b5e_442f_938b_667a26e580f3.slice/crio-6782a9c839219e93fee1885d3817af98f99a097fbfb59a4572f0762a19fb6b37 WatchSource:0}: Error finding container 6782a9c839219e93fee1885d3817af98f99a097fbfb59a4572f0762a19fb6b37: Status 404 returned error can't find the container with id 6782a9c839219e93fee1885d3817af98f99a097fbfb59a4572f0762a19fb6b37
Sep 09 03:59:11.438358 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.438335    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kdvjc"]
Sep 09 03:59:11.450715 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.450689    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:11.533247 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:11.533211    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:12.312275 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:12.312229    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lggsm" event={"ID":"5a976dc5-634b-425c-88f4-d8e2822e1911","Type":"ContainerStarted","Data":"8258c4f1628c091b77768f0a4ac45b1f87a61c1f0b6aeb36c0b6ba70e4d8118c"}
Sep 09 03:59:12.314313 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:12.314280    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8d765" event={"ID":"5e8503b7-f81e-4115-aa5d-e38becb6e75a","Type":"ContainerStarted","Data":"9111eda974a35666e05a6c4036fbed15cb8b99f77a824f474cd960d9586daa72"}
Sep 09 03:59:12.314437 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:12.314360    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8d765" event={"ID":"5e8503b7-f81e-4115-aa5d-e38becb6e75a","Type":"ContainerStarted","Data":"92ca4033c0fc346d5672a4108b9cf8b59a616f0da33cdff88624c32ec1b1b5d4"}
Sep 09 03:59:12.315596 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:12.315553    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kdvjc" event={"ID":"cfbd047c-3b5e-442f-938b-667a26e580f3","Type":"ContainerStarted","Data":"6782a9c839219e93fee1885d3817af98f99a097fbfb59a4572f0762a19fb6b37"}
Sep 09 03:59:13.419774 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.419727    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:13.460232 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.460210    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-ndc25"]
Sep 09 03:59:13.484360 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.484342    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:13.487651 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.487632    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-ndc25"]
Sep 09 03:59:13.487745 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.487733    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25"
Sep 09 03:59:13.490214 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.490195    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 03:59:13.491194 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.491174    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 03:59:13.491300 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.491205    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 03:59:13.491300 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.491218    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qpscv\""
Sep 09 03:59:13.491300 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.491183    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 03:59:13.491300 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.491183    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 03:59:13.579537 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.579482    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:13.645876 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.645833    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2aec475f-c9df-4064-a147-04f43d2f50e9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-ndc25\" (UID: \"2aec475f-c9df-4064-a147-04f43d2f50e9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25"
Sep 09 03:59:13.646015 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.645883    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5bl\" (UniqueName: \"kubernetes.io/projected/2aec475f-c9df-4064-a147-04f43d2f50e9-kube-api-access-sq5bl\") pod \"prometheus-operator-6f4946db56-ndc25\" (UID: \"2aec475f-c9df-4064-a147-04f43d2f50e9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25"
Sep 09 03:59:13.646015 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.645921    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2aec475f-c9df-4064-a147-04f43d2f50e9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-ndc25\" (UID: \"2aec475f-c9df-4064-a147-04f43d2f50e9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25"
Sep 09 03:59:13.646102 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.646015    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aec475f-c9df-4064-a147-04f43d2f50e9-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-ndc25\" (UID: \"2aec475f-c9df-4064-a147-04f43d2f50e9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25"
Sep 09 03:59:13.746597 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.746569    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2aec475f-c9df-4064-a147-04f43d2f50e9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-ndc25\" (UID: \"2aec475f-c9df-4064-a147-04f43d2f50e9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25"
Sep 09 03:59:13.746761 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.746609    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5bl\" (UniqueName: \"kubernetes.io/projected/2aec475f-c9df-4064-a147-04f43d2f50e9-kube-api-access-sq5bl\") pod \"prometheus-operator-6f4946db56-ndc25\" (UID: \"2aec475f-c9df-4064-a147-04f43d2f50e9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25"
Sep 09 03:59:13.746761 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.746634    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2aec475f-c9df-4064-a147-04f43d2f50e9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-ndc25\" (UID: \"2aec475f-c9df-4064-a147-04f43d2f50e9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25"
Sep 09 03:59:13.746761 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.746678    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aec475f-c9df-4064-a147-04f43d2f50e9-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-ndc25\" (UID: \"2aec475f-c9df-4064-a147-04f43d2f50e9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25"
Sep 09 03:59:13.747398 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.747351    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aec475f-c9df-4064-a147-04f43d2f50e9-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-ndc25\" (UID: \"2aec475f-c9df-4064-a147-04f43d2f50e9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25"
Sep 09 03:59:13.749402 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.749383    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2aec475f-c9df-4064-a147-04f43d2f50e9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-ndc25\" (UID: \"2aec475f-c9df-4064-a147-04f43d2f50e9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25"
Sep 09 03:59:13.749402 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.749391    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2aec475f-c9df-4064-a147-04f43d2f50e9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-ndc25\" (UID: \"2aec475f-c9df-4064-a147-04f43d2f50e9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25"
Sep 09 03:59:13.755911 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.755890    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5bl\" (UniqueName: \"kubernetes.io/projected/2aec475f-c9df-4064-a147-04f43d2f50e9-kube-api-access-sq5bl\") pod \"prometheus-operator-6f4946db56-ndc25\" (UID: \"2aec475f-c9df-4064-a147-04f43d2f50e9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25"
Sep 09 03:59:13.795754 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:13.795731    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25"
Sep 09 03:59:14.323755 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:14.323695    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8d765" event={"ID":"5e8503b7-f81e-4115-aa5d-e38becb6e75a","Type":"ContainerStarted","Data":"573b3ac46c6d05e6386a4f06821486d36d3626caf8693453a4573a9d5d5b735e"}
Sep 09 03:59:14.534725 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:14.534611    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-ndc25"]
Sep 09 03:59:14.538565 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:14.538529    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aec475f_c9df_4064_a147_04f43d2f50e9.slice/crio-7994535c48b6426fa45249ed26e51ef5cb2f6e2a649ee95396124fc5270c5c9f WatchSource:0}: Error finding container 7994535c48b6426fa45249ed26e51ef5cb2f6e2a649ee95396124fc5270c5c9f: Status 404 returned error can't find the container with id 7994535c48b6426fa45249ed26e51ef5cb2f6e2a649ee95396124fc5270c5c9f
Sep 09 03:59:15.327580 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:15.327557    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kdvjc" event={"ID":"cfbd047c-3b5e-442f-938b-667a26e580f3","Type":"ContainerStarted","Data":"50e24ca54da7dfed206ff519a778f13aae5a13774d04b7b9ed74383d8d2632c7"}
Sep 09 03:59:15.329245 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:15.329218    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lggsm" event={"ID":"5a976dc5-634b-425c-88f4-d8e2822e1911","Type":"ContainerStarted","Data":"32f9323e646c214b0d77d251731a89e274abcf4850692c0e0513804876511a78"}
Sep 09 03:59:15.329365 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:15.329267    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lggsm" event={"ID":"5a976dc5-634b-425c-88f4-d8e2822e1911","Type":"ContainerStarted","Data":"869ae273e6ef9c974af0a767204ebeb56e1e78efa4dd8b69c9e84dcd14a254c3"}
Sep 09 03:59:15.329365 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:15.329347    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:15.330645 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:15.330625    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25" event={"ID":"2aec475f-c9df-4064-a147-04f43d2f50e9","Type":"ContainerStarted","Data":"7994535c48b6426fa45249ed26e51ef5cb2f6e2a649ee95396124fc5270c5c9f"}
Sep 09 03:59:15.345928 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:15.345885    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kdvjc" podStartSLOduration=2.491104058 podStartE2EDuration="5.345872183s" totalImagesPullingTime="2.854768125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:15.345515989 +0000 UTC m=+54.871886699" watchObservedRunningTime="2026-09-09 03:59:15.345872183 +0000 UTC m=+54.872242883"
Sep 09 03:59:15.366142 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:15.363590    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lggsm" podStartSLOduration=2.452230134 podStartE2EDuration="5.363572556s" totalImagesPullingTime="2.911342422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:15.362146153 +0000 UTC m=+54.888516866" watchObservedRunningTime="2026-09-09 03:59:15.363572556 +0000 UTC m=+54.889943266"
Sep 09 03:59:16.336930 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:16.336890    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25" event={"ID":"2aec475f-c9df-4064-a147-04f43d2f50e9","Type":"ContainerStarted","Data":"4ff0c8edb25566d530d7aabd7f5f272ae868817447ee8d2bbb210e4c0320287e"}
Sep 09 03:59:16.337401 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:16.336998    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25" event={"ID":"2aec475f-c9df-4064-a147-04f43d2f50e9","Type":"ContainerStarted","Data":"3323a8d0f9d5d06e198e035f068f1e912bfd5edd1f52a1fcc20c88e3fbfb911d"}
Sep 09 03:59:16.341686 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:16.341655    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8d765" event={"ID":"5e8503b7-f81e-4115-aa5d-e38becb6e75a","Type":"ContainerStarted","Data":"a1f88a4effcd6ec34823fe9c2ac3385234d324ec75862a23eeb77752099883c0"}
Sep 09 03:59:16.355802 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:16.355753    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-ndc25" podStartSLOduration=1.747051964 podStartE2EDuration="3.355737781s" totalImagesPullingTime="1.608685817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:16.354601484 +0000 UTC m=+55.880972193" watchObservedRunningTime="2026-09-09 03:59:16.355737781 +0000 UTC m=+55.882108491"
Sep 09 03:59:16.378157 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:16.378106    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8d765" podStartSLOduration=2.664353281 podStartE2EDuration="6.378087526s" totalImagesPullingTime="3.713734245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:59:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:16.377275596 +0000 UTC m=+55.903646305" watchObservedRunningTime="2026-09-09 03:59:16.378087526 +0000 UTC m=+55.904458239"
Sep 09 03:59:16.406201 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:16.406146    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:17.604291 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.604260    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:17.791135 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.791105    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:17.812979 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.812955    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"]
Sep 09 03:59:17.814922 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.814907    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:17.817887 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.817868    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 03:59:17.818069 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.818057    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hj74k\""
Sep 09 03:59:17.818578 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.818560    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 03:59:17.826131 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.826113    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"]
Sep 09 03:59:17.844402 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.844379    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"]
Sep 09 03:59:17.846336 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.846323    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.848472 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.848438    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 09 03:59:17.848566 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.848471    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 09 03:59:17.848622 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.848608    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-m6ht9\""
Sep 09 03:59:17.849096 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.849082    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 09 03:59:17.855098 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.855051    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gdzk5"]
Sep 09 03:59:17.857081 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.857066    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.859727 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.859705    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 03:59:17.859870 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.859758    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 03:59:17.860085 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.860066    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 03:59:17.860085 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.860089    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rxzhr\""
Sep 09 03:59:17.860348 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.860327    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"]
Sep 09 03:59:17.862820 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.862779    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:17.874954 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.874930    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-accelerators-collector-config\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.875054 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.874957    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/317b9d62-0af8-4984-a8bf-920ffa946469-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.875054 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.874974    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93ba5076-012e-4286-b6c2-8fdda7591135-sys\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.875054 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.874999    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-textfile\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.875054 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875015    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-qwkzn\" (UID: \"30fe4953-3b75-473f-a4ec-ca7ff55ba07d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:17.875261 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875079    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drkkl\" (UniqueName: \"kubernetes.io/projected/317b9d62-0af8-4984-a8bf-920ffa946469-kube-api-access-drkkl\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.875261 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875126    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-qwkzn\" (UID: \"30fe4953-3b75-473f-a4ec-ca7ff55ba07d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:17.875261 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875154    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-tls\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.875261 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875180    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-qwkzn\" (UID: \"30fe4953-3b75-473f-a4ec-ca7ff55ba07d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:17.875261 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875205    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.875261 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875233    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/317b9d62-0af8-4984-a8bf-920ffa946469-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.875261 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875261    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93ba5076-012e-4286-b6c2-8fdda7591135-root\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.875535 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875284    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv8l2\" (UniqueName: \"kubernetes.io/projected/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-kube-api-access-tv8l2\") pod \"openshift-state-metrics-657fd6c54-qwkzn\" (UID: \"30fe4953-3b75-473f-a4ec-ca7ff55ba07d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:17.875535 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875323    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/317b9d62-0af8-4984-a8bf-920ffa946469-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.875535 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875349    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-wtmp\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.875535 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875378    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/317b9d62-0af8-4984-a8bf-920ffa946469-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.875535 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875401    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjpt7\" (UniqueName: \"kubernetes.io/projected/93ba5076-012e-4286-b6c2-8fdda7591135-kube-api-access-zjpt7\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.875535 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875425    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93ba5076-012e-4286-b6c2-8fdda7591135-metrics-client-ca\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.875535 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.875453    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/317b9d62-0af8-4984-a8bf-920ffa946469-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.937949 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.937932    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:17.975992 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.975968    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drkkl\" (UniqueName: \"kubernetes.io/projected/317b9d62-0af8-4984-a8bf-920ffa946469-kube-api-access-drkkl\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.976083 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976008    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-qwkzn\" (UID: \"30fe4953-3b75-473f-a4ec-ca7ff55ba07d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:17.976083 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976025    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-tls\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.976083 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976054    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-qwkzn\" (UID: \"30fe4953-3b75-473f-a4ec-ca7ff55ba07d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:17.976212 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976080    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.976212 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:17.976106    2553 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 09 03:59:17.976212 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976109    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/317b9d62-0af8-4984-a8bf-920ffa946469-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.976212 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976135    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93ba5076-012e-4286-b6c2-8fdda7591135-root\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.976212 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:17.976161    2553 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 03:59:17.976212 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976189    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93ba5076-012e-4286-b6c2-8fdda7591135-root\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.976212 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:17.976169    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-openshift-state-metrics-tls podName:30fe4953-3b75-473f-a4ec-ca7ff55ba07d nodeName:}" failed. No retries permitted until 2026-09-09 03:59:18.476151735 +0000 UTC m=+58.002522423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-qwkzn" (UID: "30fe4953-3b75-473f-a4ec-ca7ff55ba07d") : secret "openshift-state-metrics-tls" not found
Sep 09 03:59:17.976568 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976233    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv8l2\" (UniqueName: \"kubernetes.io/projected/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-kube-api-access-tv8l2\") pod \"openshift-state-metrics-657fd6c54-qwkzn\" (UID: \"30fe4953-3b75-473f-a4ec-ca7ff55ba07d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:17.976568 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976261    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/317b9d62-0af8-4984-a8bf-920ffa946469-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.976568 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976286    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-wtmp\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.976568 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:17.976312    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-tls podName:93ba5076-012e-4286-b6c2-8fdda7591135 nodeName:}" failed. No retries permitted until 2026-09-09 03:59:18.476292971 +0000 UTC m=+58.002663678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-tls") pod "node-exporter-gdzk5" (UID: "93ba5076-012e-4286-b6c2-8fdda7591135") : secret "node-exporter-tls" not found
Sep 09 03:59:17.976568 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976348    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/317b9d62-0af8-4984-a8bf-920ffa946469-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.976568 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976392    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjpt7\" (UniqueName: \"kubernetes.io/projected/93ba5076-012e-4286-b6c2-8fdda7591135-kube-api-access-zjpt7\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.976568 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976397    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-wtmp\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.976568 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976489    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93ba5076-012e-4286-b6c2-8fdda7591135-metrics-client-ca\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.976568 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976512    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/317b9d62-0af8-4984-a8bf-920ffa946469-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.976568 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976536    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-accelerators-collector-config\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.976568 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976563    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/317b9d62-0af8-4984-a8bf-920ffa946469-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.976921 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976590    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93ba5076-012e-4286-b6c2-8fdda7591135-sys\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.976921 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.976608    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/317b9d62-0af8-4984-a8bf-920ffa946469-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.977159 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.977137    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-accelerators-collector-config\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.977222 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.977201    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93ba5076-012e-4286-b6c2-8fdda7591135-sys\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.977222 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.977207    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/317b9d62-0af8-4984-a8bf-920ffa946469-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.977343 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.977244    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-textfile\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.977343 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.977300    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-qwkzn\" (UID: \"30fe4953-3b75-473f-a4ec-ca7ff55ba07d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:17.977343 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.977335    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93ba5076-012e-4286-b6c2-8fdda7591135-metrics-client-ca\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.977552 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.977393    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/317b9d62-0af8-4984-a8bf-920ffa946469-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.977657 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.977637    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-textfile\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.978218 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.978198    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-qwkzn\" (UID: \"30fe4953-3b75-473f-a4ec-ca7ff55ba07d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:17.979172 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.979153    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/317b9d62-0af8-4984-a8bf-920ffa946469-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.979496 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.979478    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/317b9d62-0af8-4984-a8bf-920ffa946469-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.979579 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.979504    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.979686 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.979664    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-qwkzn\" (UID: \"30fe4953-3b75-473f-a4ec-ca7ff55ba07d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:17.985654 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.985623    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drkkl\" (UniqueName: \"kubernetes.io/projected/317b9d62-0af8-4984-a8bf-920ffa946469-kube-api-access-drkkl\") pod \"kube-state-metrics-74cb57df7-dpccj\" (UID: \"317b9d62-0af8-4984-a8bf-920ffa946469\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:17.986204 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.986172    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjpt7\" (UniqueName: \"kubernetes.io/projected/93ba5076-012e-4286-b6c2-8fdda7591135-kube-api-access-zjpt7\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:17.986606 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:17.986591    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv8l2\" (UniqueName: \"kubernetes.io/projected/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-kube-api-access-tv8l2\") pod \"openshift-state-metrics-657fd6c54-qwkzn\" (UID: \"30fe4953-3b75-473f-a4ec-ca7ff55ba07d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:18.040752 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.040732    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:18.136456 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.136396    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:18.157200 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.157183    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"
Sep 09 03:59:18.238871 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.238838    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:18.343392 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.343368    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:18.432687 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.432629    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-dpccj"]
Sep 09 03:59:18.435488 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:18.435437    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317b9d62_0af8_4984_a8bf_920ffa946469.slice/crio-de94b46ed79bfb98eac23cf3ef66bbcfe268e97c0ef412a6bb219303ca04d3a5 WatchSource:0}: Error finding container de94b46ed79bfb98eac23cf3ef66bbcfe268e97c0ef412a6bb219303ca04d3a5: Status 404 returned error can't find the container with id de94b46ed79bfb98eac23cf3ef66bbcfe268e97c0ef412a6bb219303ca04d3a5
Sep 09 03:59:18.476351 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.476322    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:18.481529 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.481508    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-qwkzn\" (UID: \"30fe4953-3b75-473f-a4ec-ca7ff55ba07d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:18.481592 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.481534    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-tls\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:18.483787 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.483763    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93ba5076-012e-4286-b6c2-8fdda7591135-node-exporter-tls\") pod \"node-exporter-gdzk5\" (UID: \"93ba5076-012e-4286-b6c2-8fdda7591135\") " pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:18.483886 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.483869    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30fe4953-3b75-473f-a4ec-ca7ff55ba07d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-qwkzn\" (UID: \"30fe4953-3b75-473f-a4ec-ca7ff55ba07d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:18.723746 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.723725    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"
Sep 09 03:59:18.767400 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.767217    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gdzk5"
Sep 09 03:59:18.779899 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:18.779868    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ba5076_012e_4286_b6c2_8fdda7591135.slice/crio-63589ba438804807377f99bd50299d98fdc34529cff79bdb8260fd500b24a50f WatchSource:0}: Error finding container 63589ba438804807377f99bd50299d98fdc34529cff79bdb8260fd500b24a50f: Status 404 returned error can't find the container with id 63589ba438804807377f99bd50299d98fdc34529cff79bdb8260fd500b24a50f
Sep 09 03:59:18.922553 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.922518    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:18.996393 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:18.996333    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn"]
Sep 09 03:59:18.999456 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:18.999418    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fe4953_3b75_473f_a4ec_ca7ff55ba07d.slice/crio-5d8f394e4e2e2942df7426c3fa2e967b7774f484dcb36fb9eacc31a80b005319 WatchSource:0}: Error finding container 5d8f394e4e2e2942df7426c3fa2e967b7774f484dcb36fb9eacc31a80b005319: Status 404 returned error can't find the container with id 5d8f394e4e2e2942df7426c3fa2e967b7774f484dcb36fb9eacc31a80b005319
Sep 09 03:59:19.022233 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.022209    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:19.044074 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.044042    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:59:19.046775 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.046755    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.053019 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.052998    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 03:59:19.053167 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.053148    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6cnkf\""
Sep 09 03:59:19.057306 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.054912    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 03:59:19.057306 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.055150    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 03:59:19.057306 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.055312    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 03:59:19.057306 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.056426    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 03:59:19.057306 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.056704    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 03:59:19.057306 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.056904    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 03:59:19.057306 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.057082    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 03:59:19.057306 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.057219    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 03:59:19.068324 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.068301    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:59:19.084827 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.084800    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.084915 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.084850    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.084915 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.084877    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/362dac6f-8fcd-460c-800a-790403cfa0d7-config-out\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.084915 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.084905    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-web-config\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.085162 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.084939    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.085162 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.084982    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.085162 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.085030    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4569\" (UniqueName: \"kubernetes.io/projected/362dac6f-8fcd-460c-800a-790403cfa0d7-kube-api-access-b4569\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.085162 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.085056    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.085162 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.085083    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-config-volume\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.085162 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.085110    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/362dac6f-8fcd-460c-800a-790403cfa0d7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.085162 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.085136    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.085162 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.085165    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.085618 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.085206    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.134852 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.134824    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:19.186436 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.186361    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4569\" (UniqueName: \"kubernetes.io/projected/362dac6f-8fcd-460c-800a-790403cfa0d7-kube-api-access-b4569\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.186436 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.186427    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.186655 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.186474    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-config-volume\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.186655 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.186493    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/362dac6f-8fcd-460c-800a-790403cfa0d7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.186655 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.186512    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.186655 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.186530    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.186655 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.186557    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.186655 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.186586    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.186655 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.186607    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.186655 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.186623    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/362dac6f-8fcd-460c-800a-790403cfa0d7-config-out\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.186655 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.186639    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-web-config\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.186655 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.186656    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.187099 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.186678    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.187099 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.186961    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.187644 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:19.187576    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-trusted-ca-bundle podName:362dac6f-8fcd-460c-800a-790403cfa0d7 nodeName:}" failed. No retries permitted until 2026-09-09 03:59:19.687552404 +0000 UTC m=+59.213923115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "362dac6f-8fcd-460c-800a-790403cfa0d7") : configmap references non-existent config key: ca-bundle.crt
Sep 09 03:59:19.188152 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.188106    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.190071 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.190042    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-web-config\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.190174 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.190049    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.190427 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.190407    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.191290 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.191028    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.191290 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.191157    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.191418 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.191296    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/362dac6f-8fcd-460c-800a-790403cfa0d7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.191776 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.191752    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.191894 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.191860    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-config-volume\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.191961 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.191901    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/362dac6f-8fcd-460c-800a-790403cfa0d7-config-out\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.196189 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.196159    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4569\" (UniqueName: \"kubernetes.io/projected/362dac6f-8fcd-460c-800a-790403cfa0d7-kube-api-access-b4569\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.353210 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.353131    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj" event={"ID":"317b9d62-0af8-4984-a8bf-920ffa946469","Type":"ContainerStarted","Data":"de94b46ed79bfb98eac23cf3ef66bbcfe268e97c0ef412a6bb219303ca04d3a5"}
Sep 09 03:59:19.354780 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.354743    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gdzk5" event={"ID":"93ba5076-012e-4286-b6c2-8fdda7591135","Type":"ContainerStarted","Data":"63589ba438804807377f99bd50299d98fdc34529cff79bdb8260fd500b24a50f"}
Sep 09 03:59:19.357539 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.357516    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn" event={"ID":"30fe4953-3b75-473f-a4ec-ca7ff55ba07d","Type":"ContainerStarted","Data":"5cd8a2a26f8f9713db4ba41c69e19e330d3d7c0a14952dd265f88f48fb420e05"}
Sep 09 03:59:19.357649 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.357560    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn" event={"ID":"30fe4953-3b75-473f-a4ec-ca7ff55ba07d","Type":"ContainerStarted","Data":"128520e50a00cdc368f086843b4af57433230f9ba1811ba8bc4c7461cc2c99fa"}
Sep 09 03:59:19.357649 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.357579    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn" event={"ID":"30fe4953-3b75-473f-a4ec-ca7ff55ba07d","Type":"ContainerStarted","Data":"5d8f394e4e2e2942df7426c3fa2e967b7774f484dcb36fb9eacc31a80b005319"}
Sep 09 03:59:19.690378 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.690290    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.691169 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.691137    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.953490 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.953402    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67f97f9cd-2qngm"]
Sep 09 03:59:19.960985 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.960962    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:59:19.974488 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.974463    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67f97f9cd-2qngm"]
Sep 09 03:59:19.974600 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.974548    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:19.977091 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.977069    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 03:59:19.977218 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.977110    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bpuvi59eo0cdr\""
Sep 09 03:59:19.977312 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.977223    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 03:59:19.977406 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.977380    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 03:59:19.977662 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.977638    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 03:59:19.977662 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.977662    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rhlvj\""
Sep 09 03:59:19.977938 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.977920    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 03:59:19.993197 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.993174    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-grpc-tls\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:19.993316 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.993275    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:19.993316 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.993306    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:19.993429 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.993358    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aeaf7fa0-947d-4f08-b87c-344f6f366d40-metrics-client-ca\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:19.993429 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.993391    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:19.993555 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.993512    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-tls\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:19.993555 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.993543    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:19.993657 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:19.993575    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4cjm\" (UniqueName: \"kubernetes.io/projected/aeaf7fa0-947d-4f08-b87c-344f6f366d40-kube-api-access-h4cjm\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.094877 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.094848    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-tls\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.095044 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.094889    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.095044 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.094927    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cjm\" (UniqueName: \"kubernetes.io/projected/aeaf7fa0-947d-4f08-b87c-344f6f366d40-kube-api-access-h4cjm\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.095044 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.095009    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-grpc-tls\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.095184 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.095053    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.095184 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.095080    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.095184 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.095127    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aeaf7fa0-947d-4f08-b87c-344f6f366d40-metrics-client-ca\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.095184 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.095160    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.095943 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.095916    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aeaf7fa0-947d-4f08-b87c-344f6f366d40-metrics-client-ca\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.098298 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.098272    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.098416 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.098334    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.098600 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.098580    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.098672 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.098633    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.099370 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.099348    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-thanos-querier-tls\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.099477 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.099426    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aeaf7fa0-947d-4f08-b87c-344f6f366d40-secret-grpc-tls\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.105346 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.105321    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4cjm\" (UniqueName: \"kubernetes.io/projected/aeaf7fa0-947d-4f08-b87c-344f6f366d40-kube-api-access-h4cjm\") pod \"thanos-querier-67f97f9cd-2qngm\" (UID: \"aeaf7fa0-947d-4f08-b87c-344f6f366d40\") " pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.294267 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.294236    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:20.862861 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.862799    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:20.926194 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:20.926166    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:21.065982 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.065950    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:21.095334 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.095298    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67f97f9cd-2qngm"]
Sep 09 03:59:21.102563 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.102541    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:59:21.145796 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:21.145749    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeaf7fa0_947d_4f08_b87c_344f6f366d40.slice/crio-cfb881a1cb775389d77fc2225cc0d42c4bda4402f423f11c7ae36d4ba29df13b WatchSource:0}: Error finding container cfb881a1cb775389d77fc2225cc0d42c4bda4402f423f11c7ae36d4ba29df13b: Status 404 returned error can't find the container with id cfb881a1cb775389d77fc2225cc0d42c4bda4402f423f11c7ae36d4ba29df13b
Sep 09 03:59:21.146869 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:21.146843    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362dac6f_8fcd_460c_800a_790403cfa0d7.slice/crio-aaf59a5368b4c123a092a7e258d3aa6075f464ad873d4be72fd6be9db19f6e38 WatchSource:0}: Error finding container aaf59a5368b4c123a092a7e258d3aa6075f464ad873d4be72fd6be9db19f6e38: Status 404 returned error can't find the container with id aaf59a5368b4c123a092a7e258d3aa6075f464ad873d4be72fd6be9db19f6e38
Sep 09 03:59:21.269651 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.269620    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v45h8"
Sep 09 03:59:21.301496 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.301410    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:21.366110 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.366082    2553 generic.go:356] "Generic (PLEG): container finished" podID="93ba5076-012e-4286-b6c2-8fdda7591135" containerID="266da4f737e4b8d473e7fef3df756ef7efcf682bb93fdc71cdb8489a9ad87a32" exitCode=0
Sep 09 03:59:21.366243 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.366182    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gdzk5" event={"ID":"93ba5076-012e-4286-b6c2-8fdda7591135","Type":"ContainerDied","Data":"266da4f737e4b8d473e7fef3df756ef7efcf682bb93fdc71cdb8489a9ad87a32"}
Sep 09 03:59:21.368657 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.368633    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn" event={"ID":"30fe4953-3b75-473f-a4ec-ca7ff55ba07d","Type":"ContainerStarted","Data":"085d3edf38eee2e8b08d7883264172f49f40b64dcc0029f6b982bfb8fe9b50d2"}
Sep 09 03:59:21.369906 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.369875    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm" event={"ID":"aeaf7fa0-947d-4f08-b87c-344f6f366d40","Type":"ContainerStarted","Data":"cfb881a1cb775389d77fc2225cc0d42c4bda4402f423f11c7ae36d4ba29df13b"}
Sep 09 03:59:21.371204 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.371184    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerStarted","Data":"aaf59a5368b4c123a092a7e258d3aa6075f464ad873d4be72fd6be9db19f6e38"}
Sep 09 03:59:21.373480 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.373459    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj" event={"ID":"317b9d62-0af8-4984-a8bf-920ffa946469","Type":"ContainerStarted","Data":"29276c223534daaaaa0626bc4339fb1acbc66d4d6c3831cd122fabee1b8f0bd7"}
Sep 09 03:59:21.373555 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.373502    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj" event={"ID":"317b9d62-0af8-4984-a8bf-920ffa946469","Type":"ContainerStarted","Data":"45fc142f11bccd82d4a238653cbe38b6b9d1230406b59b108a2a849f0c7247c1"}
Sep 09 03:59:21.373555 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.373520    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj" event={"ID":"317b9d62-0af8-4984-a8bf-920ffa946469","Type":"ContainerStarted","Data":"9373c0099f78aa751084199376cdea8562f5fe5c78e0e6b20ac313206dada6bf"}
Sep 09 03:59:21.399504 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.399481    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:21.409718 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.409196    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpccj" podStartSLOduration=2.04725861 podStartE2EDuration="4.409177623s" totalImagesPullingTime="2.361919013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:21.406925558 +0000 UTC m=+60.933296268" watchObservedRunningTime="2026-09-09 03:59:21.409177623 +0000 UTC m=+60.935548332"
Sep 09 03:59:21.427191 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.427151    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-qwkzn" podStartSLOduration=2.412256716 podStartE2EDuration="4.427135459s" totalImagesPullingTime="2.014878743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:21.426658213 +0000 UTC m=+60.953028922" watchObservedRunningTime="2026-09-09 03:59:21.427135459 +0000 UTC m=+60.953506168"
Sep 09 03:59:21.459757 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.459736    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:21.560849 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:21.560792    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:22.236901 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.235969    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-69df5c8d65-28k72"]
Sep 09 03:59:22.238974 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.238183    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.241484 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.241462    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 03:59:22.241752 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.241735    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 03:59:22.241834 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.241815    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 03:59:22.241905 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.241737    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 03:59:22.242087 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.242069    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-86qvc\""
Sep 09 03:59:22.242713 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.242695    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2j6lp0j9f91hp\""
Sep 09 03:59:22.249067 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.249024    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69df5c8d65-28k72"]
Sep 09 03:59:22.315239 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.315215    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/662b2ae2-722e-4de5-87db-da25b85882d2-audit-log\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.315348 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.315250    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/662b2ae2-722e-4de5-87db-da25b85882d2-metrics-server-audit-profiles\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.315348 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.315284    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/662b2ae2-722e-4de5-87db-da25b85882d2-client-ca-bundle\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.315348 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.315335    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mvz5\" (UniqueName: \"kubernetes.io/projected/662b2ae2-722e-4de5-87db-da25b85882d2-kube-api-access-4mvz5\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.315467 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.315351    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/662b2ae2-722e-4de5-87db-da25b85882d2-secret-metrics-server-client-certs\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.315467 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.315410    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/662b2ae2-722e-4de5-87db-da25b85882d2-secret-metrics-server-tls\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.315467 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.315455    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/662b2ae2-722e-4de5-87db-da25b85882d2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.381642 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.381610    2553 generic.go:356] "Generic (PLEG): container finished" podID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerID="af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5" exitCode=0
Sep 09 03:59:22.381752 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.381701    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerDied","Data":"af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5"}
Sep 09 03:59:22.384694 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.384666    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gdzk5" event={"ID":"93ba5076-012e-4286-b6c2-8fdda7591135","Type":"ContainerStarted","Data":"bde987e1e594031c1137ea594c42293b61705783d86a5e840d17c6edc1e9972f"}
Sep 09 03:59:22.384814 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.384719    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gdzk5" event={"ID":"93ba5076-012e-4286-b6c2-8fdda7591135","Type":"ContainerStarted","Data":"264098e7d0377917e747f8f9ecc567872518530c18302f65830ef1c3eb4eb28b"}
Sep 09 03:59:22.416285 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.416228    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/662b2ae2-722e-4de5-87db-da25b85882d2-client-ca-bundle\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.417140 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.416534    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mvz5\" (UniqueName: \"kubernetes.io/projected/662b2ae2-722e-4de5-87db-da25b85882d2-kube-api-access-4mvz5\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.417140 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.416576    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/662b2ae2-722e-4de5-87db-da25b85882d2-secret-metrics-server-client-certs\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.417140 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.416898    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/662b2ae2-722e-4de5-87db-da25b85882d2-secret-metrics-server-tls\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.417140 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.416967    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/662b2ae2-722e-4de5-87db-da25b85882d2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.417140 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.417056    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/662b2ae2-722e-4de5-87db-da25b85882d2-audit-log\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.417547 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.417141    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/662b2ae2-722e-4de5-87db-da25b85882d2-metrics-server-audit-profiles\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.417719 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.417692    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/662b2ae2-722e-4de5-87db-da25b85882d2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.417953 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.417915    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/662b2ae2-722e-4de5-87db-da25b85882d2-audit-log\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.418758 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.418733    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/662b2ae2-722e-4de5-87db-da25b85882d2-metrics-server-audit-profiles\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.419558 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.419534    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/662b2ae2-722e-4de5-87db-da25b85882d2-client-ca-bundle\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.419813 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.419796    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/662b2ae2-722e-4de5-87db-da25b85882d2-secret-metrics-server-client-certs\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.420039 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.420023    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/662b2ae2-722e-4de5-87db-da25b85882d2-secret-metrics-server-tls\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.424484 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.424468    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mvz5\" (UniqueName: \"kubernetes.io/projected/662b2ae2-722e-4de5-87db-da25b85882d2-kube-api-access-4mvz5\") pod \"metrics-server-69df5c8d65-28k72\" (UID: \"662b2ae2-722e-4de5-87db-da25b85882d2\") " pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.429113 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.429050    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gdzk5" podStartSLOduration=3.308676891 podStartE2EDuration="5.429036968s" totalImagesPullingTime="2.020279127s" totalInitContainerRuntime="100.08095ms" isStatefulPod=true podCreationTimestamp="2026-09-09 03:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:22.428878478 +0000 UTC m=+61.955249188" watchObservedRunningTime="2026-09-09 03:59:22.429036968 +0000 UTC m=+61.955407677"
Sep 09 03:59:22.551279 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.551247    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:22.595549 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.595524    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:22.598544 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.598516    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn"]
Sep 09 03:59:22.600819 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.600802    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn"
Sep 09 03:59:22.603908 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.603844    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wmc7r\""
Sep 09 03:59:22.603908 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.603901    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 09 03:59:22.612631 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.612609    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn"]
Sep 09 03:59:22.618897 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.618875    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/84ebfd1d-c5ce-48af-ad63-9db8df98432c-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-rlbkn\" (UID: \"84ebfd1d-c5ce-48af-ad63-9db8df98432c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn"
Sep 09 03:59:22.619096 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.619079    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbg88\" (UniqueName: \"kubernetes.io/projected/84ebfd1d-c5ce-48af-ad63-9db8df98432c-kube-api-access-cbg88\") pod \"monitoring-plugin-85bbbbbf4-rlbkn\" (UID: \"84ebfd1d-c5ce-48af-ad63-9db8df98432c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn"
Sep 09 03:59:22.720049 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.720017    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/84ebfd1d-c5ce-48af-ad63-9db8df98432c-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-rlbkn\" (UID: \"84ebfd1d-c5ce-48af-ad63-9db8df98432c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn"
Sep 09 03:59:22.720186 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.720066    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbg88\" (UniqueName: \"kubernetes.io/projected/84ebfd1d-c5ce-48af-ad63-9db8df98432c-kube-api-access-cbg88\") pod \"monitoring-plugin-85bbbbbf4-rlbkn\" (UID: \"84ebfd1d-c5ce-48af-ad63-9db8df98432c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn"
Sep 09 03:59:22.720186 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:22.720176    2553 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 09 03:59:22.720289 ip-10-0-135-219 kubenswrapper[2553]: E0909 03:59:22.720242    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ebfd1d-c5ce-48af-ad63-9db8df98432c-monitoring-plugin-cert podName:84ebfd1d-c5ce-48af-ad63-9db8df98432c nodeName:}" failed. No retries permitted until 2026-09-09 03:59:23.22022367 +0000 UTC m=+62.746594358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/84ebfd1d-c5ce-48af-ad63-9db8df98432c-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-rlbkn" (UID: "84ebfd1d-c5ce-48af-ad63-9db8df98432c") : secret "monitoring-plugin-cert" not found
Sep 09 03:59:22.729219 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:22.729200    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbg88\" (UniqueName: \"kubernetes.io/projected/84ebfd1d-c5ce-48af-ad63-9db8df98432c-kube-api-access-cbg88\") pod \"monitoring-plugin-85bbbbbf4-rlbkn\" (UID: \"84ebfd1d-c5ce-48af-ad63-9db8df98432c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn"
Sep 09 03:59:23.030433 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.030190    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-78695676b7-ppbh6"]
Sep 09 03:59:23.032928 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.032907    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.035754 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.035728    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 03:59:23.035876 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.035728    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 03:59:23.035876 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.035785    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 03:59:23.035876 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.035813    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 03:59:23.036041 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.035891    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 03:59:23.036041 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.035901    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wkcff\""
Sep 09 03:59:23.040694 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.040674    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 03:59:23.046108 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.046090    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78695676b7-ppbh6"]
Sep 09 03:59:23.123831 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.123800    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f726385e-a889-4833-8d54-40e3856eb95b-serving-certs-ca-bundle\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.123957 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.123839    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f726385e-a889-4833-8d54-40e3856eb95b-metrics-client-ca\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.123957 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.123906    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f726385e-a889-4833-8d54-40e3856eb95b-federate-client-tls\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.124044 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.123957    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdm5c\" (UniqueName: \"kubernetes.io/projected/f726385e-a889-4833-8d54-40e3856eb95b-kube-api-access-mdm5c\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.124091 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.124045    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f726385e-a889-4833-8d54-40e3856eb95b-secret-telemeter-client\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.124091 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.124065    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f726385e-a889-4833-8d54-40e3856eb95b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.124091 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.124086    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f726385e-a889-4833-8d54-40e3856eb95b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.124236 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.124123    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f726385e-a889-4833-8d54-40e3856eb95b-telemeter-client-tls\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.226083 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.224963    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/84ebfd1d-c5ce-48af-ad63-9db8df98432c-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-rlbkn\" (UID: \"84ebfd1d-c5ce-48af-ad63-9db8df98432c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn"
Sep 09 03:59:23.226083 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.225004    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f726385e-a889-4833-8d54-40e3856eb95b-secret-telemeter-client\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.226083 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.225034    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f726385e-a889-4833-8d54-40e3856eb95b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.226083 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.225063    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f726385e-a889-4833-8d54-40e3856eb95b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.226083 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.225479    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f726385e-a889-4833-8d54-40e3856eb95b-telemeter-client-tls\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.226083 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.225550    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f726385e-a889-4833-8d54-40e3856eb95b-serving-certs-ca-bundle\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.226083 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.225578    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f726385e-a889-4833-8d54-40e3856eb95b-metrics-client-ca\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.226083 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.225630    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f726385e-a889-4833-8d54-40e3856eb95b-federate-client-tls\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.226083 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.225680    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdm5c\" (UniqueName: \"kubernetes.io/projected/f726385e-a889-4833-8d54-40e3856eb95b-kube-api-access-mdm5c\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.226083 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.225994    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f726385e-a889-4833-8d54-40e3856eb95b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.226820 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.226791    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f726385e-a889-4833-8d54-40e3856eb95b-serving-certs-ca-bundle\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.227595 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.227455    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f726385e-a889-4833-8d54-40e3856eb95b-metrics-client-ca\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.229568 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.229542    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f726385e-a889-4833-8d54-40e3856eb95b-federate-client-tls\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.230013 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.229987    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f726385e-a889-4833-8d54-40e3856eb95b-telemeter-client-tls\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.230410 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.230370    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/84ebfd1d-c5ce-48af-ad63-9db8df98432c-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-rlbkn\" (UID: \"84ebfd1d-c5ce-48af-ad63-9db8df98432c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn"
Sep 09 03:59:23.233470 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.232880    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f726385e-a889-4833-8d54-40e3856eb95b-secret-telemeter-client\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.233470 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.233200    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f726385e-a889-4833-8d54-40e3856eb95b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.233470 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.233322    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:23.235729 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.235707    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdm5c\" (UniqueName: \"kubernetes.io/projected/f726385e-a889-4833-8d54-40e3856eb95b-kube-api-access-mdm5c\") pod \"telemeter-client-78695676b7-ppbh6\" (UID: \"f726385e-a889-4833-8d54-40e3856eb95b\") " pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.320680 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.320645    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69df5c8d65-28k72"]
Sep 09 03:59:23.323345 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:23.323316    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662b2ae2_722e_4de5_87db_da25b85882d2.slice/crio-6adef7d83d0f96e567a5f7d33e12f03d71b1dbbacbcd7432c1f971c75cec4906 WatchSource:0}: Error finding container 6adef7d83d0f96e567a5f7d33e12f03d71b1dbbacbcd7432c1f971c75cec4906: Status 404 returned error can't find the container with id 6adef7d83d0f96e567a5f7d33e12f03d71b1dbbacbcd7432c1f971c75cec4906
Sep 09 03:59:23.344235 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.344214    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6"
Sep 09 03:59:23.350578 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.350550    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:23.392424 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.392392    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm" event={"ID":"aeaf7fa0-947d-4f08-b87c-344f6f366d40","Type":"ContainerStarted","Data":"4ae1ad769f0dfc23d4807d67862ec470370e831d913a42cbb04fc651fc7c4aa7"}
Sep 09 03:59:23.392554 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.392497    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm" event={"ID":"aeaf7fa0-947d-4f08-b87c-344f6f366d40","Type":"ContainerStarted","Data":"c3daeea8841c2c1afb4320bfdaf37569dbd8575b9b63d9531b7208e62ebb2aaf"}
Sep 09 03:59:23.392554 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.392541    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm" event={"ID":"aeaf7fa0-947d-4f08-b87c-344f6f366d40","Type":"ContainerStarted","Data":"5b73d8493bf9efe604d97811e46053d602f7219d623075781899c30945ca9993"}
Sep 09 03:59:23.394262 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.394061    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69df5c8d65-28k72" event={"ID":"662b2ae2-722e-4de5-87db-da25b85882d2","Type":"ContainerStarted","Data":"6adef7d83d0f96e567a5f7d33e12f03d71b1dbbacbcd7432c1f971c75cec4906"}
Sep 09 03:59:23.448961 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.448939    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:23.511520 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.511494    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn"
Sep 09 03:59:23.629090 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.628973    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78695676b7-ppbh6"]
Sep 09 03:59:23.633778 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:23.633749    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf726385e_a889_4833_8d54_40e3856eb95b.slice/crio-9739ad6546c31dec25d20f21c015bfc3b0acdc4569218c79f3912a915730f224 WatchSource:0}: Error finding container 9739ad6546c31dec25d20f21c015bfc3b0acdc4569218c79f3912a915730f224: Status 404 returned error can't find the container with id 9739ad6546c31dec25d20f21c015bfc3b0acdc4569218c79f3912a915730f224
Sep 09 03:59:23.787367 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:23.787334    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn"]
Sep 09 03:59:23.790115 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:23.790072    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ebfd1d_c5ce_48af_ad63_9db8df98432c.slice/crio-c61ceea5b3dcf1666f3e69c8d279361ce843e4d8d9e58c1ef549a21caab87a98 WatchSource:0}: Error finding container c61ceea5b3dcf1666f3e69c8d279361ce843e4d8d9e58c1ef549a21caab87a98: Status 404 returned error can't find the container with id c61ceea5b3dcf1666f3e69c8d279361ce843e4d8d9e58c1ef549a21caab87a98
Sep 09 03:59:24.204842 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.204760    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:24.212492 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.210340    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:59:24.215387 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.215364    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.218673 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.218107    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 03:59:24.218673 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.218122    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-j30aei1piflt\""
Sep 09 03:59:24.218673 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.218416    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 03:59:24.218673 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.218417    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 03:59:24.219498 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.219077    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 03:59:24.219498 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.219329    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 03:59:24.220933 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.219893    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 03:59:24.220933 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.219947    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dgndq\""
Sep 09 03:59:24.220933 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.220508    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 03:59:24.220933 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.220809    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 03:59:24.221658 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.221325    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 03:59:24.221658 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.221422    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 03:59:24.221658 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.221552    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 03:59:24.222260 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.222071    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 03:59:24.222260 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.222080    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 03:59:24.231187 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.231163    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 03:59:24.237464 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.237420    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:59:24.238016 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.237857    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-config\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.238016 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.237956    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.238016 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.238092    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.238016 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.238213    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.238016 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.238270    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.239201 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.239181    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.239327 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.239313    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.239415 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.239402    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.239548 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.239533    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.239685 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.239670    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58g27\" (UniqueName: \"kubernetes.io/projected/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-kube-api-access-58g27\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.239839 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.239815    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.239980 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.239956    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.240128 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.240114    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.240271 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.240249    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.240600 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.240584    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.240778 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.240762    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.240922 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.240907    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.241024 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.241010    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.241126 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.241112    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.241234 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.241219    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.342745 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.342711    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.342745 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.342751    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.343225 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.342790    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-config\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.343225 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.342817    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.343225 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.342840    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.343995 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.343538    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344181    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344225    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344252    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344276    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344297    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344333    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344378    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58g27\" (UniqueName: \"kubernetes.io/projected/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-kube-api-access-58g27\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344403    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344436    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344517    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344540    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344561    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344605    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344662    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.345059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.344685    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.347077 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.347051    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.350348 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.348091    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.350348 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.348140    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.350348 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.348501    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.350348 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.349512    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.350348 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.349733    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.350348 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.350146    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.350808 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.350774    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.350868 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.350824    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.351306 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.351286    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.352714 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.352682    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.353404 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.353336    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.354346 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.353843    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.354346 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.354032    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.354346 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.354311    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-config\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.355238 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.355176    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.355873 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.355845    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.359189 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.359159    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.361618 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.361600    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58g27\" (UniqueName: \"kubernetes.io/projected/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-kube-api-access-58g27\") pod \"prometheus-k8s-0\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.399596 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.399563    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn" event={"ID":"84ebfd1d-c5ce-48af-ad63-9db8df98432c","Type":"ContainerStarted","Data":"c61ceea5b3dcf1666f3e69c8d279361ce843e4d8d9e58c1ef549a21caab87a98"}
Sep 09 03:59:24.401466 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.401421    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6" event={"ID":"f726385e-a889-4833-8d54-40e3856eb95b","Type":"ContainerStarted","Data":"9739ad6546c31dec25d20f21c015bfc3b0acdc4569218c79f3912a915730f224"}
Sep 09 03:59:24.538837 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.538795    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:24.539155 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.539136    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:24.644249 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.643927    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:24.699476 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.699368    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:24.813104 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.813027    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:24.996951 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:24.996857    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:25.194114 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:25.194051    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:25.249154 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:25.249115    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:25.343824 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:25.343798    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lggsm"
Sep 09 03:59:25.399769 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:25.399704    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:25.499646 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:25.499611    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:25.608832 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:25.608801    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:26.012505 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.012386    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:26.110900 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.110654    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:26.214022 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.213880    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:59:26.215167 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.215016    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:26.217599 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:26.217572    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bb08fa_77d3_4157_9fdd_b38d7dd17527.slice/crio-23267dcdd537b73775a9ada3431c307241f8179018d60003c0283c1e96642405 WatchSource:0}: Error finding container 23267dcdd537b73775a9ada3431c307241f8179018d60003c0283c1e96642405: Status 404 returned error can't find the container with id 23267dcdd537b73775a9ada3431c307241f8179018d60003c0283c1e96642405
Sep 09 03:59:26.412616 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.412520    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69df5c8d65-28k72" event={"ID":"662b2ae2-722e-4de5-87db-da25b85882d2","Type":"ContainerStarted","Data":"0d282806a00f1e62f1430b3a4a200f78f86df3bce5fdab95128fd91f604bf3ac"}
Sep 09 03:59:26.414246 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.414222    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn" event={"ID":"84ebfd1d-c5ce-48af-ad63-9db8df98432c","Type":"ContainerStarted","Data":"1979959373bb18ec97be8d3a993ce06324b415e090dbd25c3c9d940ab73ea924"}
Sep 09 03:59:26.414476 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.414341    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn"
Sep 09 03:59:26.418327 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.418273    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6" event={"ID":"f726385e-a889-4833-8d54-40e3856eb95b","Type":"ContainerStarted","Data":"4bd84fdf630f3c1fec6d68101289273e03997e892aa0c0a7a5227549b6ae5c60"}
Sep 09 03:59:26.418327 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.418321    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6" event={"ID":"f726385e-a889-4833-8d54-40e3856eb95b","Type":"ContainerStarted","Data":"4531fec9007e7bfdc2279c4ac61b834a65a2c96c91355f2ee8a04a84e390fd62"}
Sep 09 03:59:26.418516 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.418352    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6" event={"ID":"f726385e-a889-4833-8d54-40e3856eb95b","Type":"ContainerStarted","Data":"f0ee751965a518f14179dde88434685864fe9c741872ff708babbded0588dccc"}
Sep 09 03:59:26.420731 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.420713    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn"
Sep 09 03:59:26.420990 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.420967    2553 generic.go:356] "Generic (PLEG): container finished" podID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerID="6783542f8ee00c96f9d6ea16b49a8555d9696aa73178e334d6449f6626b1217b" exitCode=0
Sep 09 03:59:26.421075 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.421050    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerDied","Data":"6783542f8ee00c96f9d6ea16b49a8555d9696aa73178e334d6449f6626b1217b"}
Sep 09 03:59:26.421140 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.421126    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerStarted","Data":"23267dcdd537b73775a9ada3431c307241f8179018d60003c0283c1e96642405"}
Sep 09 03:59:26.424897 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.424876    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm" event={"ID":"aeaf7fa0-947d-4f08-b87c-344f6f366d40","Type":"ContainerStarted","Data":"b5f3f29527c586702bb2de5fb7dc8c898890c62e2c05e039f40ebeb67891e571"}
Sep 09 03:59:26.424964 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.424947    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm" event={"ID":"aeaf7fa0-947d-4f08-b87c-344f6f366d40","Type":"ContainerStarted","Data":"3c3f30b5bee4ff6f771b7459cbf46f58318aa6166c5b02bd7069bf9dfb8896bc"}
Sep 09 03:59:26.425009 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.424996    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm" event={"ID":"aeaf7fa0-947d-4f08-b87c-344f6f366d40","Type":"ContainerStarted","Data":"2fb69badb97019ea8eaf17825f3e03325aacc2aaa2acc427949d82ad6a7ae9a9"}
Sep 09 03:59:26.425585 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.425568    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:26.429295 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.429277    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerStarted","Data":"45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a"}
Sep 09 03:59:26.429397 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.429356    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerStarted","Data":"6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2"}
Sep 09 03:59:26.429499 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.429401    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerStarted","Data":"8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd"}
Sep 09 03:59:26.429499 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.429469    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerStarted","Data":"a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc"}
Sep 09 03:59:26.442079 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.441987    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-69df5c8d65-28k72" podStartSLOduration=1.815665418 podStartE2EDuration="4.441972383s" totalImagesPullingTime="2.626306965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:26.441944649 +0000 UTC m=+65.968315364" watchObservedRunningTime="2026-09-09 03:59:26.441972383 +0000 UTC m=+65.968343094"
Sep 09 03:59:26.477284 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.477255    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:26.531007 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.530963    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm" podStartSLOduration=3.391602045 podStartE2EDuration="7.530949516s" totalImagesPullingTime="4.139347471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:26.529948646 +0000 UTC m=+66.056319365" watchObservedRunningTime="2026-09-09 03:59:26.530949516 +0000 UTC m=+66.057320227"
Sep 09 03:59:26.575701 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.575661    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-78695676b7-ppbh6" podStartSLOduration=1.249136726 podStartE2EDuration="3.575646327s" totalImagesPullingTime="2.326509601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:26.575341351 +0000 UTC m=+66.101712065" watchObservedRunningTime="2026-09-09 03:59:26.575646327 +0000 UTC m=+66.102017036"
Sep 09 03:59:26.595727 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.595709    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:26.601947 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.601905    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rlbkn" podStartSLOduration=2.437139786 podStartE2EDuration="4.601886444s" totalImagesPullingTime="2.164746658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:26.601382371 +0000 UTC m=+66.127753081" watchObservedRunningTime="2026-09-09 03:59:26.601886444 +0000 UTC m=+66.128257157"
Sep 09 03:59:26.671611 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.671580    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs\") pod \"network-metrics-daemon-4x7vw\" (UID: \"9579406a-2f97-456a-8d1a-bb8db3d24ab6\") " pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:59:26.674635 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.674619    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 03:59:26.684229 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.684212    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9579406a-2f97-456a-8d1a-bb8db3d24ab6-metrics-certs\") pod \"network-metrics-daemon-4x7vw\" (UID: \"9579406a-2f97-456a-8d1a-bb8db3d24ab6\") " pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:59:26.772036 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.772002    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:59:26.774601 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.774586    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 03:59:26.785075 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.785056    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db-original-pull-secret\") pod \"global-pull-secret-syncer-kpwhr\" (UID: \"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db\") " pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:59:26.872609 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.872584    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfpcx\" (UniqueName: \"kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx\") pod \"network-check-target-njsng\" (UID: \"ccb3f1f6-8987-454c-8505-4adc83697891\") " pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:59:26.875235 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.875220    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 03:59:26.885833 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.885819    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 03:59:26.896079 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.896057    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfpcx\" (UniqueName: \"kubernetes.io/projected/ccb3f1f6-8987-454c-8505-4adc83697891-kube-api-access-pfpcx\") pod \"network-check-target-njsng\" (UID: \"ccb3f1f6-8987-454c-8505-4adc83697891\") " pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:59:26.900267 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.900253    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-krf5h\""
Sep 09 03:59:26.902219 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.902205    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpwhr"
Sep 09 03:59:26.908419 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:26.908406    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4x7vw"
Sep 09 03:59:27.195495 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:27.195404    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-slrsq\""
Sep 09 03:59:27.203828 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:27.203569    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:59:27.210263 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:27.210241    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kpwhr"]
Sep 09 03:59:27.212714 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:27.212533    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d64662_bc9c_4fd3_b0ee_0f4b2e21e3db.slice/crio-8faa189b102bacc2a047301d44b0e459ba481c00fd400d3580a1c2f7588cc15b WatchSource:0}: Error finding container 8faa189b102bacc2a047301d44b0e459ba481c00fd400d3580a1c2f7588cc15b: Status 404 returned error can't find the container with id 8faa189b102bacc2a047301d44b0e459ba481c00fd400d3580a1c2f7588cc15b
Sep 09 03:59:27.257983 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:27.257938    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4x7vw"]
Sep 09 03:59:27.261849 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:27.261817    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9579406a_2f97_456a_8d1a_bb8db3d24ab6.slice/crio-306bac9260c9620868272662e23334d9822b7472efdd9faaf06f4cc9210b0a7a WatchSource:0}: Error finding container 306bac9260c9620868272662e23334d9822b7472efdd9faaf06f4cc9210b0a7a: Status 404 returned error can't find the container with id 306bac9260c9620868272662e23334d9822b7472efdd9faaf06f4cc9210b0a7a
Sep 09 03:59:27.445059 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:27.445023    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerStarted","Data":"7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a"}
Sep 09 03:59:27.445576 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:27.445105    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerStarted","Data":"139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c"}
Sep 09 03:59:27.446787 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:27.446763    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4x7vw" event={"ID":"9579406a-2f97-456a-8d1a-bb8db3d24ab6","Type":"ContainerStarted","Data":"306bac9260c9620868272662e23334d9822b7472efdd9faaf06f4cc9210b0a7a"}
Sep 09 03:59:27.448089 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:27.448067    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kpwhr" event={"ID":"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db","Type":"ContainerStarted","Data":"8faa189b102bacc2a047301d44b0e459ba481c00fd400d3580a1c2f7588cc15b"}
Sep 09 03:59:27.475069 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:27.475025    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.352607691 podStartE2EDuration="8.47501085s" totalImagesPullingTime="4.101263801s" totalInitContainerRuntime="21.139358ms" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:27.474005384 +0000 UTC m=+67.000376096" watchObservedRunningTime="2026-09-09 03:59:27.47501085 +0000 UTC m=+67.001381559"
Sep 09 03:59:27.479055 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:27.479033    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:27.497296 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:27.497258    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njsng"]
Sep 09 03:59:27.506987 ip-10-0-135-219 kubenswrapper[2553]: W0909 03:59:27.502485    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb3f1f6_8987_454c_8505_4adc83697891.slice/crio-0c728dd30ad2bd9061e353c8765b882e6ba6f38cbb3104adba165ef7ea60ee11 WatchSource:0}: Error finding container 0c728dd30ad2bd9061e353c8765b882e6ba6f38cbb3104adba165ef7ea60ee11: Status 404 returned error can't find the container with id 0c728dd30ad2bd9061e353c8765b882e6ba6f38cbb3104adba165ef7ea60ee11
Sep 09 03:59:27.579792 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:27.579755    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:28.276404 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:28.276335    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:28.341231 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:28.340579    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:28.457079 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:28.457044    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njsng" event={"ID":"ccb3f1f6-8987-454c-8505-4adc83697891","Type":"ContainerStarted","Data":"0c728dd30ad2bd9061e353c8765b882e6ba6f38cbb3104adba165ef7ea60ee11"}
Sep 09 03:59:28.471045 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:28.471010    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67f97f9cd-2qngm"
Sep 09 03:59:29.469032 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:29.466533    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4x7vw" event={"ID":"9579406a-2f97-456a-8d1a-bb8db3d24ab6","Type":"ContainerStarted","Data":"35a0eca0c0d4779dc6a28a304dd06199172a3a96ba467a1198995de497fe2ec3"}
Sep 09 03:59:29.469032 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:29.466576    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4x7vw" event={"ID":"9579406a-2f97-456a-8d1a-bb8db3d24ab6","Type":"ContainerStarted","Data":"3adfa1a780d7f2c99846b7441db2106a9ed974089de2efa1fd6975c03e072eb5"}
Sep 09 03:59:29.626129 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:29.626074    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:29.725031 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:29.724959    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:31.523239 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:31.523196    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:31.589347 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:31.589295    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:33.267035 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.266832    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:33.403096 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.403052    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:33.491863 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.491832    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerStarted","Data":"56dbd8cb03fb053dc7b407882df4ac6e3ebc3bd8b976c047931f462b244b3136"}
Sep 09 03:59:33.491997 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.491930    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerStarted","Data":"9ef6dc103b9559c6d9b406f0186e381f36dc0428d56a992934aeeefac3ee5000"}
Sep 09 03:59:33.491997 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.491961    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerStarted","Data":"4a38fb6b677f9808f4f837cd70542ca5fe30a0a3950f9d7a50a84c2fb7a7bbac"}
Sep 09 03:59:33.491997 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.491986    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerStarted","Data":"e50bac675739f277934463499cd6859c0453b0738f346e4a93998890b4233fa2"}
Sep 09 03:59:33.492224 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.492011    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerStarted","Data":"efde60555f58f562881c724dae411a5e9e141cc8e8815993f3728ed071ce1286"}
Sep 09 03:59:33.492224 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.492034    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerStarted","Data":"5f53b3137b6a1e0732aa50d0ba46d5bb2b8b81db482e80857ecbec248dd57a94"}
Sep 09 03:59:33.493895 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.493872    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njsng" event={"ID":"ccb3f1f6-8987-454c-8505-4adc83697891","Type":"ContainerStarted","Data":"b69896ec34a14beb6379f88b226b3cf377e49fa43e1312c1d422360bc00d0d76"}
Sep 09 03:59:33.494083 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.494054    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 03:59:33.495702 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.495685    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kpwhr" event={"ID":"a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db","Type":"ContainerStarted","Data":"3e9481b6884c9e61ac6dbdf38b04be69bb92007e54a41899f21c1fc6532b9a78"}
Sep 09 03:59:33.529981 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.529851    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4x7vw" podStartSLOduration=71.200321933 podStartE2EDuration="1m12.529835646s" totalImagesPullingTime="1.329513713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:29.496199135 +0000 UTC m=+69.022569844" watchObservedRunningTime="2026-09-09 03:59:33.529835646 +0000 UTC m=+73.056206357"
Sep 09 03:59:33.530939 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.530622    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.900467237 podStartE2EDuration="9.530612193s" totalImagesPullingTime="6.603616551s" totalInitContainerRuntime="26.528405ms" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:33.527253115 +0000 UTC m=+73.053623823" watchObservedRunningTime="2026-09-09 03:59:33.530612193 +0000 UTC m=+73.056982903"
Sep 09 03:59:33.546946 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.546914    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-njsng" podStartSLOduration=67.013851071 podStartE2EDuration="1m12.546902091s" totalImagesPullingTime="5.53305102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:33.545066141 +0000 UTC m=+73.071436852" watchObservedRunningTime="2026-09-09 03:59:33.546902091 +0000 UTC m=+73.073272799"
Sep 09 03:59:33.564995 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.564954    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kpwhr" podStartSLOduration=66.740891066 podStartE2EDuration="1m12.564941892s" totalImagesPullingTime="5.824050826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:33.563600104 +0000 UTC m=+73.089970817" watchObservedRunningTime="2026-09-09 03:59:33.564941892 +0000 UTC m=+73.091312600"
Sep 09 03:59:33.751420 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.751396    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:33.851342 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.851290    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:33.908433 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:33.908414    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:34.539282 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:34.539252    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:59:37.291671 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:37.291637    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:37.397964 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:37.397941    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:39.634851 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:39.634818    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:39.735193 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:39.735169    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:40.643329 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:40.643294    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:40.702277 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:40.702246    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:41.594589 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:41.594558    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:41.652131 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:41.652110    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:42.552097 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:42.551993    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:42.552097 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:42.552054    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 03:59:43.272843 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:43.272809    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:46.644177 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:46.644141    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:47.843828 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:47.843780    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:47.950268 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:47.950236    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:51.843660 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:51.843626    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:51.898284 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:51.898251    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:52.247272 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:52.247241    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:52.306187 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:52.306157    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:52.843521 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:52.843495    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:52.903798 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:52.903774    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:56.468881 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:56.468850    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 03:59:56.522641 ip-10-0-135-219 kubenswrapper[2553]: I0909 03:59:56.522620    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:00:02.556304 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:02.556276    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 04:00:02.560043 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:02.560017    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-69df5c8d65-28k72"
Sep 09 04:00:04.502096 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:04.502065    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-njsng"
Sep 09 04:00:06.183370 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:06.183331    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:00:24.539695 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:24.539665    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:24.559695 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:24.559673    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:24.716518 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:24.716495    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:34.014138 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:34.014091    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:00:34.072791 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:34.072766    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:00:38.269985 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.269956    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 04:00:38.270376 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.270235    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="alertmanager" containerID="cri-o://a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc" gracePeriod=120
Sep 09 04:00:38.270376 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.270288    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="kube-rbac-proxy-metric" containerID="cri-o://7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a" gracePeriod=120
Sep 09 04:00:38.270376 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.270303    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="config-reloader" containerID="cri-o://8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd" gracePeriod=120
Sep 09 04:00:38.270577 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.270352    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="prom-label-proxy" containerID="cri-o://139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c" gracePeriod=120
Sep 09 04:00:38.270577 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.270429    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="kube-rbac-proxy" containerID="cri-o://45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a" gracePeriod=120
Sep 09 04:00:38.270684 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.270595    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="kube-rbac-proxy-web" containerID="cri-o://6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2" gracePeriod=120
Sep 09 04:00:38.755609 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.755580    2553 generic.go:356] "Generic (PLEG): container finished" podID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerID="139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c" exitCode=0
Sep 09 04:00:38.755609 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.755606    2553 generic.go:356] "Generic (PLEG): container finished" podID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerID="45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a" exitCode=0
Sep 09 04:00:38.755609 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.755614    2553 generic.go:356] "Generic (PLEG): container finished" podID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerID="8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd" exitCode=0
Sep 09 04:00:38.755813 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.755619    2553 generic.go:356] "Generic (PLEG): container finished" podID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerID="a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc" exitCode=0
Sep 09 04:00:38.755813 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.755647    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerDied","Data":"139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c"}
Sep 09 04:00:38.755813 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.755709    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerDied","Data":"45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a"}
Sep 09 04:00:38.755813 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.755734    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerDied","Data":"8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd"}
Sep 09 04:00:38.755813 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:38.755758    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerDied","Data":"a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc"}
Sep 09 04:00:39.499820 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.499798    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.540346 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.540321    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-metrics-client-ca\") pod \"362dac6f-8fcd-460c-800a-790403cfa0d7\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") "
Sep 09 04:00:39.540507 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.540359    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy-web\") pod \"362dac6f-8fcd-460c-800a-790403cfa0d7\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") "
Sep 09 04:00:39.540507 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.540383    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-main-tls\") pod \"362dac6f-8fcd-460c-800a-790403cfa0d7\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") "
Sep 09 04:00:39.540507 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.540401    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-config-volume\") pod \"362dac6f-8fcd-460c-800a-790403cfa0d7\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") "
Sep 09 04:00:39.540507 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.540436    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4569\" (UniqueName: \"kubernetes.io/projected/362dac6f-8fcd-460c-800a-790403cfa0d7-kube-api-access-b4569\") pod \"362dac6f-8fcd-460c-800a-790403cfa0d7\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") "
Sep 09 04:00:39.540507 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.540503    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy\") pod \"362dac6f-8fcd-460c-800a-790403cfa0d7\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") "
Sep 09 04:00:39.540769 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.540541    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/362dac6f-8fcd-460c-800a-790403cfa0d7-tls-assets\") pod \"362dac6f-8fcd-460c-800a-790403cfa0d7\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") "
Sep 09 04:00:39.540769 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.540575    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-trusted-ca-bundle\") pod \"362dac6f-8fcd-460c-800a-790403cfa0d7\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") "
Sep 09 04:00:39.540769 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.540609    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-cluster-tls-config\") pod \"362dac6f-8fcd-460c-800a-790403cfa0d7\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") "
Sep 09 04:00:39.540769 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.540638    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/362dac6f-8fcd-460c-800a-790403cfa0d7-config-out\") pod \"362dac6f-8fcd-460c-800a-790403cfa0d7\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") "
Sep 09 04:00:39.540769 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.540702    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"362dac6f-8fcd-460c-800a-790403cfa0d7\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") "
Sep 09 04:00:39.540769 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.540708    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "362dac6f-8fcd-460c-800a-790403cfa0d7" (UID: "362dac6f-8fcd-460c-800a-790403cfa0d7"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:00:39.540769 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.540743    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-main-db\") pod \"362dac6f-8fcd-460c-800a-790403cfa0d7\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") "
Sep 09 04:00:39.541257 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.540779    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-web-config\") pod \"362dac6f-8fcd-460c-800a-790403cfa0d7\" (UID: \"362dac6f-8fcd-460c-800a-790403cfa0d7\") "
Sep 09 04:00:39.541257 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.541047    2553 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-metrics-client-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:39.542362 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.542326    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "362dac6f-8fcd-460c-800a-790403cfa0d7" (UID: "362dac6f-8fcd-460c-800a-790403cfa0d7"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:00:39.543035 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.542953    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "362dac6f-8fcd-460c-800a-790403cfa0d7" (UID: "362dac6f-8fcd-460c-800a-790403cfa0d7"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:00:39.543722 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.543693    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "362dac6f-8fcd-460c-800a-790403cfa0d7" (UID: "362dac6f-8fcd-460c-800a-790403cfa0d7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:39.544362 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.544310    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "362dac6f-8fcd-460c-800a-790403cfa0d7" (UID: "362dac6f-8fcd-460c-800a-790403cfa0d7"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:39.544796 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.544758    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "362dac6f-8fcd-460c-800a-790403cfa0d7" (UID: "362dac6f-8fcd-460c-800a-790403cfa0d7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:39.546212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.546186    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362dac6f-8fcd-460c-800a-790403cfa0d7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "362dac6f-8fcd-460c-800a-790403cfa0d7" (UID: "362dac6f-8fcd-460c-800a-790403cfa0d7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:00:39.546566 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.546541    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362dac6f-8fcd-460c-800a-790403cfa0d7-config-out" (OuterVolumeSpecName: "config-out") pod "362dac6f-8fcd-460c-800a-790403cfa0d7" (UID: "362dac6f-8fcd-460c-800a-790403cfa0d7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:00:39.546923 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.546884    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-config-volume" (OuterVolumeSpecName: "config-volume") pod "362dac6f-8fcd-460c-800a-790403cfa0d7" (UID: "362dac6f-8fcd-460c-800a-790403cfa0d7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:39.547469 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.547412    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362dac6f-8fcd-460c-800a-790403cfa0d7-kube-api-access-b4569" (OuterVolumeSpecName: "kube-api-access-b4569") pod "362dac6f-8fcd-460c-800a-790403cfa0d7" (UID: "362dac6f-8fcd-460c-800a-790403cfa0d7"). InnerVolumeSpecName "kube-api-access-b4569". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:00:39.547799 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.547772    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "362dac6f-8fcd-460c-800a-790403cfa0d7" (UID: "362dac6f-8fcd-460c-800a-790403cfa0d7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:39.552139 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.552114    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "362dac6f-8fcd-460c-800a-790403cfa0d7" (UID: "362dac6f-8fcd-460c-800a-790403cfa0d7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:39.559121 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.559098    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-web-config" (OuterVolumeSpecName: "web-config") pod "362dac6f-8fcd-460c-800a-790403cfa0d7" (UID: "362dac6f-8fcd-460c-800a-790403cfa0d7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:39.642341 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.642290    2553 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:39.642341 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.642311    2553 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-cluster-tls-config\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:39.642341 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.642320    2553 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/362dac6f-8fcd-460c-800a-790403cfa0d7-config-out\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:39.642341 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.642330    2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:39.642341 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.642338    2553 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/362dac6f-8fcd-460c-800a-790403cfa0d7-alertmanager-main-db\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:39.642602 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.642346    2553 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-web-config\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:39.642602 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.642354    2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:39.642602 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.642362    2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-main-tls\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:39.642602 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.642370    2553 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:39.642602 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.642391    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4569\" (UniqueName: \"kubernetes.io/projected/362dac6f-8fcd-460c-800a-790403cfa0d7-kube-api-access-b4569\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:39.642602 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.642399    2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/362dac6f-8fcd-460c-800a-790403cfa0d7-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:39.642602 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.642407    2553 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/362dac6f-8fcd-460c-800a-790403cfa0d7-tls-assets\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:39.762364 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.762335    2553 generic.go:356] "Generic (PLEG): container finished" podID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerID="7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a" exitCode=0
Sep 09 04:00:39.762364 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.762356    2553 generic.go:356] "Generic (PLEG): container finished" podID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerID="6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2" exitCode=0
Sep 09 04:00:39.762514 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.762396    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerDied","Data":"7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a"}
Sep 09 04:00:39.762514 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.762465    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerDied","Data":"6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2"}
Sep 09 04:00:39.762514 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.762494    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"362dac6f-8fcd-460c-800a-790403cfa0d7","Type":"ContainerDied","Data":"aaf59a5368b4c123a092a7e258d3aa6075f464ad873d4be72fd6be9db19f6e38"}
Sep 09 04:00:39.762514 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.762494    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.762635 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.762523    2553 scope.go:117] "RemoveContainer" containerID="139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c"
Sep 09 04:00:39.771410 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.771394    2553 scope.go:117] "RemoveContainer" containerID="7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a"
Sep 09 04:00:39.778130 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.778115    2553 scope.go:117] "RemoveContainer" containerID="45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a"
Sep 09 04:00:39.784899 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.784883    2553 scope.go:117] "RemoveContainer" containerID="6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2"
Sep 09 04:00:39.786845 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.786804    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 04:00:39.793467 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.793426    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 04:00:39.793983 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.793968    2553 scope.go:117] "RemoveContainer" containerID="8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd"
Sep 09 04:00:39.800734 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.800718    2553 scope.go:117] "RemoveContainer" containerID="a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc"
Sep 09 04:00:39.807295 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.807279    2553 scope.go:117] "RemoveContainer" containerID="af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5"
Sep 09 04:00:39.814131 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.814116    2553 scope.go:117] "RemoveContainer" containerID="139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c"
Sep 09 04:00:39.814409 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:00:39.814392    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c\": container with ID starting with 139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c not found: ID does not exist" containerID="139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c"
Sep 09 04:00:39.814531 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.814429    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c"} err="failed to get container status \"139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c\": rpc error: code = NotFound desc = could not find container \"139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c\": container with ID starting with 139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c not found: ID does not exist"
Sep 09 04:00:39.814581 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.814534    2553 scope.go:117] "RemoveContainer" containerID="7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a"
Sep 09 04:00:39.814784 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:00:39.814765    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a\": container with ID starting with 7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a not found: ID does not exist" containerID="7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a"
Sep 09 04:00:39.814848 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.814794    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a"} err="failed to get container status \"7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a\": rpc error: code = NotFound desc = could not find container \"7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a\": container with ID starting with 7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a not found: ID does not exist"
Sep 09 04:00:39.814848 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.814818    2553 scope.go:117] "RemoveContainer" containerID="45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a"
Sep 09 04:00:39.815062 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:00:39.815047    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a\": container with ID starting with 45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a not found: ID does not exist" containerID="45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a"
Sep 09 04:00:39.815098 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.815068    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a"} err="failed to get container status \"45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a\": rpc error: code = NotFound desc = could not find container \"45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a\": container with ID starting with 45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a not found: ID does not exist"
Sep 09 04:00:39.815098 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.815080    2553 scope.go:117] "RemoveContainer" containerID="6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2"
Sep 09 04:00:39.815283 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:00:39.815267    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2\": container with ID starting with 6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2 not found: ID does not exist" containerID="6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2"
Sep 09 04:00:39.815349 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.815289    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2"} err="failed to get container status \"6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2\": rpc error: code = NotFound desc = could not find container \"6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2\": container with ID starting with 6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2 not found: ID does not exist"
Sep 09 04:00:39.815349 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.815308    2553 scope.go:117] "RemoveContainer" containerID="8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd"
Sep 09 04:00:39.815554 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:00:39.815539    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd\": container with ID starting with 8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd not found: ID does not exist" containerID="8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd"
Sep 09 04:00:39.815596 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.815558    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd"} err="failed to get container status \"8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd\": rpc error: code = NotFound desc = could not find container \"8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd\": container with ID starting with 8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd not found: ID does not exist"
Sep 09 04:00:39.815596 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.815571    2553 scope.go:117] "RemoveContainer" containerID="a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc"
Sep 09 04:00:39.815776 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:00:39.815760    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc\": container with ID starting with a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc not found: ID does not exist" containerID="a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc"
Sep 09 04:00:39.815841 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.815784    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc"} err="failed to get container status \"a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc\": rpc error: code = NotFound desc = could not find container \"a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc\": container with ID starting with a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc not found: ID does not exist"
Sep 09 04:00:39.815841 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.815801    2553 scope.go:117] "RemoveContainer" containerID="af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5"
Sep 09 04:00:39.816027 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:00:39.816014    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5\": container with ID starting with af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5 not found: ID does not exist" containerID="af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5"
Sep 09 04:00:39.816063 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.816030    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5"} err="failed to get container status \"af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5\": rpc error: code = NotFound desc = could not find container \"af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5\": container with ID starting with af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5 not found: ID does not exist"
Sep 09 04:00:39.816063 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.816042    2553 scope.go:117] "RemoveContainer" containerID="139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c"
Sep 09 04:00:39.816266 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.816248    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c"} err="failed to get container status \"139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c\": rpc error: code = NotFound desc = could not find container \"139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c\": container with ID starting with 139f58801863545ad339d50d5fc0db9f006104e5fd2228ed443e7ae7c456cc5c not found: ID does not exist"
Sep 09 04:00:39.816313 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.816274    2553 scope.go:117] "RemoveContainer" containerID="7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a"
Sep 09 04:00:39.816567 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.816543    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a"} err="failed to get container status \"7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a\": rpc error: code = NotFound desc = could not find container \"7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a\": container with ID starting with 7bb096a97bcd1302583894ce4774ad11544518c25d7ebc8f57ce9a9b2fb0a50a not found: ID does not exist"
Sep 09 04:00:39.816644 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.816568    2553 scope.go:117] "RemoveContainer" containerID="45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a"
Sep 09 04:00:39.816825 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.816808    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a"} err="failed to get container status \"45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a\": rpc error: code = NotFound desc = could not find container \"45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a\": container with ID starting with 45b9c523061c9ae1ad5d30e8a3ca4ce88aa21ac0fc5a81b7b2d97feaebcc901a not found: ID does not exist"
Sep 09 04:00:39.816881 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.816826    2553 scope.go:117] "RemoveContainer" containerID="6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2"
Sep 09 04:00:39.817059 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.817042    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2"} err="failed to get container status \"6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2\": rpc error: code = NotFound desc = could not find container \"6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2\": container with ID starting with 6040237277df0d8bd4ce465b85719946ed19c259d330af9aebed30655504d1c2 not found: ID does not exist"
Sep 09 04:00:39.817111 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.817058    2553 scope.go:117] "RemoveContainer" containerID="8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd"
Sep 09 04:00:39.817266 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.817249    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd"} err="failed to get container status \"8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd\": rpc error: code = NotFound desc = could not find container \"8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd\": container with ID starting with 8517ff5107d0cd9cdc2feff49c939f269486bc4eb4da055e7bd51b823169fadd not found: ID does not exist"
Sep 09 04:00:39.817307 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.817268    2553 scope.go:117] "RemoveContainer" containerID="a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc"
Sep 09 04:00:39.817551 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.817535    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc"} err="failed to get container status \"a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc\": rpc error: code = NotFound desc = could not find container \"a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc\": container with ID starting with a5312569de82145f7b087bbf8cdb8f2c3abe49198d3c6d2be2af7ac8093d65cc not found: ID does not exist"
Sep 09 04:00:39.817615 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.817552    2553 scope.go:117] "RemoveContainer" containerID="af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5"
Sep 09 04:00:39.817774 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.817755    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5"} err="failed to get container status \"af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5\": rpc error: code = NotFound desc = could not find container \"af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5\": container with ID starting with af6b1bc5aaab9707199d34d28cc60623fa8e041abb6491fd5bc54bee0097e3a5 not found: ID does not exist"
Sep 09 04:00:39.822383 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822364    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 04:00:39.822705 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822692    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="prom-label-proxy"
Sep 09 04:00:39.822758 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822707    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="prom-label-proxy"
Sep 09 04:00:39.822758 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822716    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="init-config-reloader"
Sep 09 04:00:39.822758 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822723    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="init-config-reloader"
Sep 09 04:00:39.822758 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822747    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="config-reloader"
Sep 09 04:00:39.822758 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822753    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="config-reloader"
Sep 09 04:00:39.822895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822761    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="alertmanager"
Sep 09 04:00:39.822895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822766    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="alertmanager"
Sep 09 04:00:39.822895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822773    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="kube-rbac-proxy-web"
Sep 09 04:00:39.822895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822778    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="kube-rbac-proxy-web"
Sep 09 04:00:39.822895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822785    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="kube-rbac-proxy-metric"
Sep 09 04:00:39.822895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822790    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="kube-rbac-proxy-metric"
Sep 09 04:00:39.822895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822798    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="kube-rbac-proxy"
Sep 09 04:00:39.822895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822802    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="kube-rbac-proxy"
Sep 09 04:00:39.822895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822842    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="config-reloader"
Sep 09 04:00:39.822895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822849    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="kube-rbac-proxy"
Sep 09 04:00:39.822895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822855    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="kube-rbac-proxy-metric"
Sep 09 04:00:39.822895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822861    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="alertmanager"
Sep 09 04:00:39.822895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822867    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="kube-rbac-proxy-web"
Sep 09 04:00:39.822895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.822873    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" containerName="prom-label-proxy"
Sep 09 04:00:39.826157 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.826141    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.828569 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.828553    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 04:00:39.828704 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.828688    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 04:00:39.828788 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.828742    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 04:00:39.829040 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.829015    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 04:00:39.829125 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.829076    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 04:00:39.829125 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.829093    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6cnkf\""
Sep 09 04:00:39.829269 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.829128    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 04:00:39.829269 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.829091    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 04:00:39.829269 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.829014    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 04:00:39.834169 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.834149    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 04:00:39.843654 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.843628    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.843752 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.843691    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/956fd919-b132-466b-9ddf-451c34ab9a22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.843752 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.843738    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/956fd919-b132-466b-9ddf-451c34ab9a22-config-out\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.843844 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.843786    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956fd919-b132-466b-9ddf-451c34ab9a22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.843844 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.843830    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.843947 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.843856    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.843947 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.843881    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/956fd919-b132-466b-9ddf-451c34ab9a22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.843947 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.843921    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.844044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.843961    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-config-volume\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.844044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.843987    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-web-config\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.844044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.844008    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/956fd919-b132-466b-9ddf-451c34ab9a22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.844044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.844041    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crwxf\" (UniqueName: \"kubernetes.io/projected/956fd919-b132-466b-9ddf-451c34ab9a22-kube-api-access-crwxf\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.844179 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.844120    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.844598 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.844583    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 04:00:39.945421 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.945345    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956fd919-b132-466b-9ddf-451c34ab9a22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.945421 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.945398    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.945421 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.945422    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.945650 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.945553    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/956fd919-b132-466b-9ddf-451c34ab9a22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.945650 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.945591    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.945650 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.945624    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-config-volume\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.945796 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.945651    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-web-config\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.945796 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.945678    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/956fd919-b132-466b-9ddf-451c34ab9a22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.945796 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.945714    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crwxf\" (UniqueName: \"kubernetes.io/projected/956fd919-b132-466b-9ddf-451c34ab9a22-kube-api-access-crwxf\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.945796 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.945744    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.945796 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.945780    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.946043 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.945846    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/956fd919-b132-466b-9ddf-451c34ab9a22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.946043 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.945873    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/956fd919-b132-466b-9ddf-451c34ab9a22-config-out\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.946043 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.945938    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/956fd919-b132-466b-9ddf-451c34ab9a22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.946564 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.946540    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956fd919-b132-466b-9ddf-451c34ab9a22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.946909 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.946865    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/956fd919-b132-466b-9ddf-451c34ab9a22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.948569 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.948525    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/956fd919-b132-466b-9ddf-451c34ab9a22-config-out\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.948650 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.948573    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.948877 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.948673    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.948877 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.948857    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.948877 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.948872    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.949419 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.949397    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-web-config\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.949670 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.949647    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/956fd919-b132-466b-9ddf-451c34ab9a22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.949814 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.949797    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.950389 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.950372    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/956fd919-b132-466b-9ddf-451c34ab9a22-config-volume\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:39.954861 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:39.954844    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crwxf\" (UniqueName: \"kubernetes.io/projected/956fd919-b132-466b-9ddf-451c34ab9a22-kube-api-access-crwxf\") pod \"alertmanager-main-0\" (UID: \"956fd919-b132-466b-9ddf-451c34ab9a22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:40.135079 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:40.135057    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 04:00:40.415689 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:40.415666    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 04:00:40.417184 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:00:40.417156    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956fd919_b132_466b_9ddf_451c34ab9a22.slice/crio-78776972dad56cf3e0a1e6d35474de5d61592161f186559f1fe4db7dbc35303a WatchSource:0}: Error finding container 78776972dad56cf3e0a1e6d35474de5d61592161f186559f1fe4db7dbc35303a: Status 404 returned error can't find the container with id 78776972dad56cf3e0a1e6d35474de5d61592161f186559f1fe4db7dbc35303a
Sep 09 04:00:40.768666 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:40.768638    2553 generic.go:356] "Generic (PLEG): container finished" podID="956fd919-b132-466b-9ddf-451c34ab9a22" containerID="fa5a74f0880ce6bc601bb0eec60e702140fcc29b172f1994f65bfe7e71b4aa3c" exitCode=0
Sep 09 04:00:40.768991 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:40.768708    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956fd919-b132-466b-9ddf-451c34ab9a22","Type":"ContainerDied","Data":"fa5a74f0880ce6bc601bb0eec60e702140fcc29b172f1994f65bfe7e71b4aa3c"}
Sep 09 04:00:40.768991 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:40.768760    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956fd919-b132-466b-9ddf-451c34ab9a22","Type":"ContainerStarted","Data":"78776972dad56cf3e0a1e6d35474de5d61592161f186559f1fe4db7dbc35303a"}
Sep 09 04:00:40.990414 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:40.990388    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="362dac6f-8fcd-460c-800a-790403cfa0d7" path="/var/lib/kubelet/pods/362dac6f-8fcd-460c-800a-790403cfa0d7/volumes"
Sep 09 04:00:41.776823 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:41.776790    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956fd919-b132-466b-9ddf-451c34ab9a22","Type":"ContainerStarted","Data":"03b9b249d07b23544c7f46fd573897b2d4bfe057bb295ac6ee55e5d5f92df930"}
Sep 09 04:00:41.777243 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:41.776852    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956fd919-b132-466b-9ddf-451c34ab9a22","Type":"ContainerStarted","Data":"cc21d52b6d4816514a7a38e534fc122c735c4f5c203184e2534b6d68b2ca2c4e"}
Sep 09 04:00:41.777243 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:41.776877    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956fd919-b132-466b-9ddf-451c34ab9a22","Type":"ContainerStarted","Data":"9b34893f9809b5abfbaa4849241cc91a0754d35cf96a5493b27be18d26d1944d"}
Sep 09 04:00:41.777243 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:41.776901    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956fd919-b132-466b-9ddf-451c34ab9a22","Type":"ContainerStarted","Data":"3d917dc6c601656036f2952d4ed951b82443dfba78ebaa3fd322f392f523762c"}
Sep 09 04:00:41.777243 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:41.776924    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956fd919-b132-466b-9ddf-451c34ab9a22","Type":"ContainerStarted","Data":"bcd37e35a05ff8858d31eda187c260408afe8d3c05ba7a7ae725577ff6aecaa5"}
Sep 09 04:00:41.777243 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:41.776948    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956fd919-b132-466b-9ddf-451c34ab9a22","Type":"ContainerStarted","Data":"4519dbcb7b6ee530808304d64546f0efcb95dd060323e6525be21fee03951bdd"}
Sep 09 04:00:41.807166 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:41.807108    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.793744439 podStartE2EDuration="2.807090427s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.345988ms" isStatefulPod=false podCreationTimestamp="2026-09-09 04:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:00:41.804858325 +0000 UTC m=+141.331229099" watchObservedRunningTime="2026-09-09 04:00:41.807090427 +0000 UTC m=+141.333461136"
Sep 09 04:00:42.497909 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.497876    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 04:00:42.498225 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.498173    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="prometheus" containerID="cri-o://5f53b3137b6a1e0732aa50d0ba46d5bb2b8b81db482e80857ecbec248dd57a94" gracePeriod=600
Sep 09 04:00:42.498463 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.498195    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="config-reloader" containerID="cri-o://efde60555f58f562881c724dae411a5e9e141cc8e8815993f3728ed071ce1286" gracePeriod=600
Sep 09 04:00:42.498549 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.498212    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="thanos-sidecar" containerID="cri-o://e50bac675739f277934463499cd6859c0453b0738f346e4a93998890b4233fa2" gracePeriod=600
Sep 09 04:00:42.498616 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.498196    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="kube-rbac-proxy" containerID="cri-o://9ef6dc103b9559c6d9b406f0186e381f36dc0428d56a992934aeeefac3ee5000" gracePeriod=600
Sep 09 04:00:42.498662 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.498223    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="kube-rbac-proxy-web" containerID="cri-o://4a38fb6b677f9808f4f837cd70542ca5fe30a0a3950f9d7a50a84c2fb7a7bbac" gracePeriod=600
Sep 09 04:00:42.498729 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.498223    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="kube-rbac-proxy-thanos" containerID="cri-o://56dbd8cb03fb053dc7b407882df4ac6e3ebc3bd8b976c047931f462b244b3136" gracePeriod=600
Sep 09 04:00:42.784807 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.784728    2553 generic.go:356] "Generic (PLEG): container finished" podID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerID="56dbd8cb03fb053dc7b407882df4ac6e3ebc3bd8b976c047931f462b244b3136" exitCode=0
Sep 09 04:00:42.784807 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.784753    2553 generic.go:356] "Generic (PLEG): container finished" podID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerID="9ef6dc103b9559c6d9b406f0186e381f36dc0428d56a992934aeeefac3ee5000" exitCode=0
Sep 09 04:00:42.784807 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.784760    2553 generic.go:356] "Generic (PLEG): container finished" podID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerID="e50bac675739f277934463499cd6859c0453b0738f346e4a93998890b4233fa2" exitCode=0
Sep 09 04:00:42.784807 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.784766    2553 generic.go:356] "Generic (PLEG): container finished" podID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerID="efde60555f58f562881c724dae411a5e9e141cc8e8815993f3728ed071ce1286" exitCode=0
Sep 09 04:00:42.784807 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.784771    2553 generic.go:356] "Generic (PLEG): container finished" podID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerID="5f53b3137b6a1e0732aa50d0ba46d5bb2b8b81db482e80857ecbec248dd57a94" exitCode=0
Sep 09 04:00:42.784807 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.784797    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerDied","Data":"56dbd8cb03fb053dc7b407882df4ac6e3ebc3bd8b976c047931f462b244b3136"}
Sep 09 04:00:42.785330 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.784884    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerDied","Data":"9ef6dc103b9559c6d9b406f0186e381f36dc0428d56a992934aeeefac3ee5000"}
Sep 09 04:00:42.785330 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.784913    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerDied","Data":"e50bac675739f277934463499cd6859c0453b0738f346e4a93998890b4233fa2"}
Sep 09 04:00:42.785330 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.784938    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerDied","Data":"efde60555f58f562881c724dae411a5e9e141cc8e8815993f3728ed071ce1286"}
Sep 09 04:00:42.785330 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:42.784982    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerDied","Data":"5f53b3137b6a1e0732aa50d0ba46d5bb2b8b81db482e80857ecbec248dd57a94"}
Sep 09 04:00:43.731437 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.731414    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.774233 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774202    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.774337 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774255    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-config-out\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.774337 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774288    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.774337 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774317    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-kube-rbac-proxy\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.774526 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774364    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-metrics-client-ca\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.774526 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774429    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-metrics-client-certs\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.774526 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774478    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-db\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.774526 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774501    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-1\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.774719 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774527    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-serving-certs-ca-bundle\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.774719 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774558    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-tls-assets\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.774719 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774585    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-grpc-tls\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.774719 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774609    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-tls\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.774719 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774631    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58g27\" (UniqueName: \"kubernetes.io/projected/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-kube-api-access-58g27\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.774719 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774666    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-0\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.774719 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774699    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-web-config\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.775093 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774722    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-thanos-prometheus-http-client-file\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.775093 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774747    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-config\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.775093 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774769    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-2\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.775093 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774795    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-trusted-ca-bundle\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.775093 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.774832    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-kubelet-serving-ca-bundle\") pod \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\" (UID: \"b1bb08fa-77d3-4157-9fdd-b38d7dd17527\") "
Sep 09 04:00:43.775333 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.775102    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:00:43.775391 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.775375    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:00:43.776633 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.776409    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:00:43.777146 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.777099    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:00:43.777689 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.777389    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:43.777689 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.777393    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:00:43.777887 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.777705    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:00:43.777957 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.777905    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:00:43.778298 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.778243    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:00:43.779251 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.779226    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-config-out" (OuterVolumeSpecName: "config-out") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:00:43.779342 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.779306    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:43.779909 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.779876    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:00:43.781753 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.780986    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:43.781753 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.781359    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-kube-api-access-58g27" (OuterVolumeSpecName: "kube-api-access-58g27") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "kube-api-access-58g27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:00:43.781963 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.781926    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:43.782437 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.782076    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:43.782538 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.782466    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:43.783340 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.783317    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:43.783740 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.783542    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-config" (OuterVolumeSpecName: "config") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:43.797399 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.797373    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-web-config" (OuterVolumeSpecName: "web-config") pod "b1bb08fa-77d3-4157-9fdd-b38d7dd17527" (UID: "b1bb08fa-77d3-4157-9fdd-b38d7dd17527"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:43.797762 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.797508    2553 generic.go:356] "Generic (PLEG): container finished" podID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerID="4a38fb6b677f9808f4f837cd70542ca5fe30a0a3950f9d7a50a84c2fb7a7bbac" exitCode=0
Sep 09 04:00:43.797762 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.797590    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerDied","Data":"4a38fb6b677f9808f4f837cd70542ca5fe30a0a3950f9d7a50a84c2fb7a7bbac"}
Sep 09 04:00:43.797762 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.797649    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bb08fa-77d3-4157-9fdd-b38d7dd17527","Type":"ContainerDied","Data":"23267dcdd537b73775a9ada3431c307241f8179018d60003c0283c1e96642405"}
Sep 09 04:00:43.797762 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.797678    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.797953 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.797683    2553 scope.go:117] "RemoveContainer" containerID="56dbd8cb03fb053dc7b407882df4ac6e3ebc3bd8b976c047931f462b244b3136"
Sep 09 04:00:43.806530 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.806490    2553 scope.go:117] "RemoveContainer" containerID="9ef6dc103b9559c6d9b406f0186e381f36dc0428d56a992934aeeefac3ee5000"
Sep 09 04:00:43.813501 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.813431    2553 scope.go:117] "RemoveContainer" containerID="4a38fb6b677f9808f4f837cd70542ca5fe30a0a3950f9d7a50a84c2fb7a7bbac"
Sep 09 04:00:43.819869 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.819850    2553 scope.go:117] "RemoveContainer" containerID="e50bac675739f277934463499cd6859c0453b0738f346e4a93998890b4233fa2"
Sep 09 04:00:43.823079 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.823059    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 04:00:43.827750 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.827704    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 04:00:43.828675 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.828659    2553 scope.go:117] "RemoveContainer" containerID="efde60555f58f562881c724dae411a5e9e141cc8e8815993f3728ed071ce1286"
Sep 09 04:00:43.836624 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.836609    2553 scope.go:117] "RemoveContainer" containerID="5f53b3137b6a1e0732aa50d0ba46d5bb2b8b81db482e80857ecbec248dd57a94"
Sep 09 04:00:43.845074 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.845055    2553 scope.go:117] "RemoveContainer" containerID="6783542f8ee00c96f9d6ea16b49a8555d9696aa73178e334d6449f6626b1217b"
Sep 09 04:00:43.852099 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.852083    2553 scope.go:117] "RemoveContainer" containerID="56dbd8cb03fb053dc7b407882df4ac6e3ebc3bd8b976c047931f462b244b3136"
Sep 09 04:00:43.852345 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:00:43.852329    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56dbd8cb03fb053dc7b407882df4ac6e3ebc3bd8b976c047931f462b244b3136\": container with ID starting with 56dbd8cb03fb053dc7b407882df4ac6e3ebc3bd8b976c047931f462b244b3136 not found: ID does not exist" containerID="56dbd8cb03fb053dc7b407882df4ac6e3ebc3bd8b976c047931f462b244b3136"
Sep 09 04:00:43.852392 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.852353    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56dbd8cb03fb053dc7b407882df4ac6e3ebc3bd8b976c047931f462b244b3136"} err="failed to get container status \"56dbd8cb03fb053dc7b407882df4ac6e3ebc3bd8b976c047931f462b244b3136\": rpc error: code = NotFound desc = could not find container \"56dbd8cb03fb053dc7b407882df4ac6e3ebc3bd8b976c047931f462b244b3136\": container with ID starting with 56dbd8cb03fb053dc7b407882df4ac6e3ebc3bd8b976c047931f462b244b3136 not found: ID does not exist"
Sep 09 04:00:43.852392 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.852369    2553 scope.go:117] "RemoveContainer" containerID="9ef6dc103b9559c6d9b406f0186e381f36dc0428d56a992934aeeefac3ee5000"
Sep 09 04:00:43.852635 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:00:43.852622    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef6dc103b9559c6d9b406f0186e381f36dc0428d56a992934aeeefac3ee5000\": container with ID starting with 9ef6dc103b9559c6d9b406f0186e381f36dc0428d56a992934aeeefac3ee5000 not found: ID does not exist" containerID="9ef6dc103b9559c6d9b406f0186e381f36dc0428d56a992934aeeefac3ee5000"
Sep 09 04:00:43.852680 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.852639    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef6dc103b9559c6d9b406f0186e381f36dc0428d56a992934aeeefac3ee5000"} err="failed to get container status \"9ef6dc103b9559c6d9b406f0186e381f36dc0428d56a992934aeeefac3ee5000\": rpc error: code = NotFound desc = could not find container \"9ef6dc103b9559c6d9b406f0186e381f36dc0428d56a992934aeeefac3ee5000\": container with ID starting with 9ef6dc103b9559c6d9b406f0186e381f36dc0428d56a992934aeeefac3ee5000 not found: ID does not exist"
Sep 09 04:00:43.852680 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.852651    2553 scope.go:117] "RemoveContainer" containerID="4a38fb6b677f9808f4f837cd70542ca5fe30a0a3950f9d7a50a84c2fb7a7bbac"
Sep 09 04:00:43.852894 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:00:43.852878    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a38fb6b677f9808f4f837cd70542ca5fe30a0a3950f9d7a50a84c2fb7a7bbac\": container with ID starting with 4a38fb6b677f9808f4f837cd70542ca5fe30a0a3950f9d7a50a84c2fb7a7bbac not found: ID does not exist" containerID="4a38fb6b677f9808f4f837cd70542ca5fe30a0a3950f9d7a50a84c2fb7a7bbac"
Sep 09 04:00:43.852941 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.852896    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a38fb6b677f9808f4f837cd70542ca5fe30a0a3950f9d7a50a84c2fb7a7bbac"} err="failed to get container status \"4a38fb6b677f9808f4f837cd70542ca5fe30a0a3950f9d7a50a84c2fb7a7bbac\": rpc error: code = NotFound desc = could not find container \"4a38fb6b677f9808f4f837cd70542ca5fe30a0a3950f9d7a50a84c2fb7a7bbac\": container with ID starting with 4a38fb6b677f9808f4f837cd70542ca5fe30a0a3950f9d7a50a84c2fb7a7bbac not found: ID does not exist"
Sep 09 04:00:43.852941 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.852916    2553 scope.go:117] "RemoveContainer" containerID="e50bac675739f277934463499cd6859c0453b0738f346e4a93998890b4233fa2"
Sep 09 04:00:43.853175 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:00:43.853158    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50bac675739f277934463499cd6859c0453b0738f346e4a93998890b4233fa2\": container with ID starting with e50bac675739f277934463499cd6859c0453b0738f346e4a93998890b4233fa2 not found: ID does not exist" containerID="e50bac675739f277934463499cd6859c0453b0738f346e4a93998890b4233fa2"
Sep 09 04:00:43.853211 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.853190    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50bac675739f277934463499cd6859c0453b0738f346e4a93998890b4233fa2"} err="failed to get container status \"e50bac675739f277934463499cd6859c0453b0738f346e4a93998890b4233fa2\": rpc error: code = NotFound desc = could not find container \"e50bac675739f277934463499cd6859c0453b0738f346e4a93998890b4233fa2\": container with ID starting with e50bac675739f277934463499cd6859c0453b0738f346e4a93998890b4233fa2 not found: ID does not exist"
Sep 09 04:00:43.853243 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.853210    2553 scope.go:117] "RemoveContainer" containerID="efde60555f58f562881c724dae411a5e9e141cc8e8815993f3728ed071ce1286"
Sep 09 04:00:43.853465 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:00:43.853429    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efde60555f58f562881c724dae411a5e9e141cc8e8815993f3728ed071ce1286\": container with ID starting with efde60555f58f562881c724dae411a5e9e141cc8e8815993f3728ed071ce1286 not found: ID does not exist" containerID="efde60555f58f562881c724dae411a5e9e141cc8e8815993f3728ed071ce1286"
Sep 09 04:00:43.853522 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.853466    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efde60555f58f562881c724dae411a5e9e141cc8e8815993f3728ed071ce1286"} err="failed to get container status \"efde60555f58f562881c724dae411a5e9e141cc8e8815993f3728ed071ce1286\": rpc error: code = NotFound desc = could not find container \"efde60555f58f562881c724dae411a5e9e141cc8e8815993f3728ed071ce1286\": container with ID starting with efde60555f58f562881c724dae411a5e9e141cc8e8815993f3728ed071ce1286 not found: ID does not exist"
Sep 09 04:00:43.853522 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.853481    2553 scope.go:117] "RemoveContainer" containerID="5f53b3137b6a1e0732aa50d0ba46d5bb2b8b81db482e80857ecbec248dd57a94"
Sep 09 04:00:43.853700 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:00:43.853684    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f53b3137b6a1e0732aa50d0ba46d5bb2b8b81db482e80857ecbec248dd57a94\": container with ID starting with 5f53b3137b6a1e0732aa50d0ba46d5bb2b8b81db482e80857ecbec248dd57a94 not found: ID does not exist" containerID="5f53b3137b6a1e0732aa50d0ba46d5bb2b8b81db482e80857ecbec248dd57a94"
Sep 09 04:00:43.853739 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.853704    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f53b3137b6a1e0732aa50d0ba46d5bb2b8b81db482e80857ecbec248dd57a94"} err="failed to get container status \"5f53b3137b6a1e0732aa50d0ba46d5bb2b8b81db482e80857ecbec248dd57a94\": rpc error: code = NotFound desc = could not find container \"5f53b3137b6a1e0732aa50d0ba46d5bb2b8b81db482e80857ecbec248dd57a94\": container with ID starting with 5f53b3137b6a1e0732aa50d0ba46d5bb2b8b81db482e80857ecbec248dd57a94 not found: ID does not exist"
Sep 09 04:00:43.853739 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.853717    2553 scope.go:117] "RemoveContainer" containerID="6783542f8ee00c96f9d6ea16b49a8555d9696aa73178e334d6449f6626b1217b"
Sep 09 04:00:43.853941 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:00:43.853926    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6783542f8ee00c96f9d6ea16b49a8555d9696aa73178e334d6449f6626b1217b\": container with ID starting with 6783542f8ee00c96f9d6ea16b49a8555d9696aa73178e334d6449f6626b1217b not found: ID does not exist" containerID="6783542f8ee00c96f9d6ea16b49a8555d9696aa73178e334d6449f6626b1217b"
Sep 09 04:00:43.853989 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.853943    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6783542f8ee00c96f9d6ea16b49a8555d9696aa73178e334d6449f6626b1217b"} err="failed to get container status \"6783542f8ee00c96f9d6ea16b49a8555d9696aa73178e334d6449f6626b1217b\": rpc error: code = NotFound desc = could not find container \"6783542f8ee00c96f9d6ea16b49a8555d9696aa73178e334d6449f6626b1217b\": container with ID starting with 6783542f8ee00c96f9d6ea16b49a8555d9696aa73178e334d6449f6626b1217b not found: ID does not exist"
Sep 09 04:00:43.856032 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856014    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 04:00:43.856295 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856284    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="init-config-reloader"
Sep 09 04:00:43.856340 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856297    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="init-config-reloader"
Sep 09 04:00:43.856340 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856305    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="thanos-sidecar"
Sep 09 04:00:43.856340 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856310    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="thanos-sidecar"
Sep 09 04:00:43.856340 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856321    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="config-reloader"
Sep 09 04:00:43.856340 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856327    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="config-reloader"
Sep 09 04:00:43.856340 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856334    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="prometheus"
Sep 09 04:00:43.856340 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856339    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="prometheus"
Sep 09 04:00:43.856550 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856346    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="kube-rbac-proxy"
Sep 09 04:00:43.856550 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856350    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="kube-rbac-proxy"
Sep 09 04:00:43.856550 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856356    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="kube-rbac-proxy-thanos"
Sep 09 04:00:43.856550 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856360    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="kube-rbac-proxy-thanos"
Sep 09 04:00:43.856550 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856374    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="kube-rbac-proxy-web"
Sep 09 04:00:43.856550 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856379    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="kube-rbac-proxy-web"
Sep 09 04:00:43.856550 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856422    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="kube-rbac-proxy-web"
Sep 09 04:00:43.856550 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856432    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="kube-rbac-proxy"
Sep 09 04:00:43.856550 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856454    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="prometheus"
Sep 09 04:00:43.856550 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856461    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="kube-rbac-proxy-thanos"
Sep 09 04:00:43.856550 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856469    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="config-reloader"
Sep 09 04:00:43.856550 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.856475    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" containerName="thanos-sidecar"
Sep 09 04:00:43.860239 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.860223    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.863436 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.863339    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 04:00:43.863436 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.863360    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 04:00:43.863436 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.863391    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 04:00:43.865391 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.865370    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 04:00:43.865391 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.865383    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 04:00:43.865634 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.865500    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dgndq\""
Sep 09 04:00:43.865634 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.865384    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 04:00:43.865634 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.865502    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 04:00:43.865814 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.865632    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 04:00:43.865814 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.865696    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-j30aei1piflt\""
Sep 09 04:00:43.865814 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.865709    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 04:00:43.865955 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.865904    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 04:00:43.866108 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.866093    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 04:00:43.866304 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.866285    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 04:00:43.868341 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.868322    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 04:00:43.869904 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.869768    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 04:00:43.874277 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.874259    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 04:00:43.875566 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.875547    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.875666 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.875579    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd6d0785-4895-40ce-a916-5ca5c7ba6101-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.875666 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.875599    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.875768 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.875664    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.875825 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.875765    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.875825 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.875809    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.875924 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.875846    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.875924 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.875873    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.875924 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.875899    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.876062 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.875926    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.876062 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.875951    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.876062 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.875977    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.876062 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876015    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd7xh\" (UniqueName: \"kubernetes.io/projected/cd6d0785-4895-40ce-a916-5ca5c7ba6101-kube-api-access-kd7xh\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.876062 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876052    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.876222 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876084    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.876222 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876113    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-config\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.876222 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876144    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.876222 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876160    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd6d0785-4895-40ce-a916-5ca5c7ba6101-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.876222 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876188    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.876222 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876202    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876243    2553 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-web-config\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876252    2553 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876260    2553 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-config\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876268    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876276    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876285    2553 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876294    2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876304    2553 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-config-out\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876313    2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876321    2553 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-kube-rbac-proxy\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876331    2553 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-metrics-client-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876340    2553 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-metrics-client-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876348    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-db\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876355    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876364    2553 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876371    2553 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-tls-assets\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876379    2553 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-grpc-tls\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876386    2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876393    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58g27\" (UniqueName: \"kubernetes.io/projected/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-kube-api-access-58g27\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.876905 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.876401    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1bb08fa-77d3-4157-9fdd-b38d7dd17527-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:00:43.976993 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.976969    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977099 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.976996    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd6d0785-4895-40ce-a916-5ca5c7ba6101-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977099 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977014    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977099 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977028    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977099 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977045    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977099 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977088    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd6d0785-4895-40ce-a916-5ca5c7ba6101-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977279 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977127    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977279 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977165    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977279 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977186    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977279 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977204    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977279 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977230    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977279 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977252    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977279 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977276    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977663 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977301    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977663 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977337    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977663 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977384    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977663 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977412    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd7xh\" (UniqueName: \"kubernetes.io/projected/cd6d0785-4895-40ce-a916-5ca5c7ba6101-kube-api-access-kd7xh\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977663 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977476    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977663 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977504    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.977663 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977536    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-config\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.978158 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.977888    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.978304 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.978277    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.979920 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.979239    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.980845 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.980050    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd6d0785-4895-40ce-a916-5ca5c7ba6101-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.980845 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.980071    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd6d0785-4895-40ce-a916-5ca5c7ba6101-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.980845 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.980430    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.980845 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.980630    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.980845 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.980688    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.980845 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.980803    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.981375 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.980863    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.981375 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.980883    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.981857 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.981831    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.981971 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.981945    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.982231 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.982209    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-config\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.982854 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.982813    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.982979 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.982947    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.983052 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.982984    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd6d0785-4895-40ce-a916-5ca5c7ba6101-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.984503 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.983619    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.984503 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.983674    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd6d0785-4895-40ce-a916-5ca5c7ba6101-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:43.988588 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:43.988567    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd7xh\" (UniqueName: \"kubernetes.io/projected/cd6d0785-4895-40ce-a916-5ca5c7ba6101-kube-api-access-kd7xh\") pod \"prometheus-k8s-0\" (UID: \"cd6d0785-4895-40ce-a916-5ca5c7ba6101\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:44.172793 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:44.172740    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:44.450897 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:44.450876    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 04:00:44.452583 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:00:44.452559    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6d0785_4895_40ce_a916_5ca5c7ba6101.slice/crio-b686b3ff5e09375c756547db858174cf633b13f80dfa6d89e841e88f7a64a7df WatchSource:0}: Error finding container b686b3ff5e09375c756547db858174cf633b13f80dfa6d89e841e88f7a64a7df: Status 404 returned error can't find the container with id b686b3ff5e09375c756547db858174cf633b13f80dfa6d89e841e88f7a64a7df
Sep 09 04:00:44.803336 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:44.803306    2553 generic.go:356] "Generic (PLEG): container finished" podID="cd6d0785-4895-40ce-a916-5ca5c7ba6101" containerID="62c59948822918011d8fc5c95563cce4a093810341160f36d057c9a26071693c" exitCode=0
Sep 09 04:00:44.803641 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:44.803354    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd6d0785-4895-40ce-a916-5ca5c7ba6101","Type":"ContainerDied","Data":"62c59948822918011d8fc5c95563cce4a093810341160f36d057c9a26071693c"}
Sep 09 04:00:44.803641 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:44.803408    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd6d0785-4895-40ce-a916-5ca5c7ba6101","Type":"ContainerStarted","Data":"b686b3ff5e09375c756547db858174cf633b13f80dfa6d89e841e88f7a64a7df"}
Sep 09 04:00:44.994669 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:44.994631    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1bb08fa-77d3-4157-9fdd-b38d7dd17527" path="/var/lib/kubelet/pods/b1bb08fa-77d3-4157-9fdd-b38d7dd17527/volumes"
Sep 09 04:00:45.811120 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:45.811088    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd6d0785-4895-40ce-a916-5ca5c7ba6101","Type":"ContainerStarted","Data":"38ee70e567bf27db081d8c72fa6381d472e0e2a9b3773cfc6b602e56085973aa"}
Sep 09 04:00:45.811518 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:45.811146    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd6d0785-4895-40ce-a916-5ca5c7ba6101","Type":"ContainerStarted","Data":"387bef1ae7d604dc29e7bf77939196a50f00e9c902f4a4a8b3beba105d0c87ae"}
Sep 09 04:00:45.811518 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:45.811172    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd6d0785-4895-40ce-a916-5ca5c7ba6101","Type":"ContainerStarted","Data":"a2792ef6f1888817af5c0e7bb510d02b31dd069989a66ee9c7c351520b361f40"}
Sep 09 04:00:45.811518 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:45.811197    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd6d0785-4895-40ce-a916-5ca5c7ba6101","Type":"ContainerStarted","Data":"3e36e3e059594f4674db33cdb12b7579b12b9bbce1ee40a30ea0e69a3c95a990"}
Sep 09 04:00:45.811518 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:45.811221    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd6d0785-4895-40ce-a916-5ca5c7ba6101","Type":"ContainerStarted","Data":"d291135d7cdde63c851b7f7db516dbd9ae4be8535225e221d2883c79d8e343b1"}
Sep 09 04:00:45.811518 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:45.811241    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd6d0785-4895-40ce-a916-5ca5c7ba6101","Type":"ContainerStarted","Data":"11d5ca533f871ff835cb768bd3e49f37a2bc72e8e37c120012560cd87b7364d7"}
Sep 09 04:00:45.842404 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:45.842361    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.824074266 podStartE2EDuration="2.842344992s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.270726ms" isStatefulPod=false podCreationTimestamp="2026-09-09 04:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:00:45.839881992 +0000 UTC m=+145.366252723" watchObservedRunningTime="2026-09-09 04:00:45.842344992 +0000 UTC m=+145.368715704"
Sep 09 04:00:45.944753 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:45.944732    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:00:46.022972 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:46.022949    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:00:49.173515 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:49.173483    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:00:49.406582 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:49.406546    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:00:49.462689 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:49.462626    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:00:56.478819 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:56.478785    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:00:56.585131 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:00:56.585099    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:01:00.433591 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:01:00.433555    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:01:00.612484 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:01:00.612437    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:01:00.666995 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:01:00.666964    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:01:40.484488 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:01:40.484433    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:01:40.534911 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:01:40.534884    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:01:44.173253 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:01:44.173220    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:01:44.188920 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:01:44.188898    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:01:45.060116 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:01:45.060089    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 04:02:00.915123 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:00.915096    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:02.311361 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:02.311327    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:04.516802 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:04.516766    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:04.618824 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:04.618791    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:06.130667 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:06.130634    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:06.230007 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:06.229970    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:06.331166 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:06.331135    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:09.267152 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:09.267117    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:09.366195 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:09.366174    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:11.317710 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:11.317680    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:11.381567 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:11.381544    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:12.912473 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:12.912434    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cgd7g"]
Sep 09 04:02:12.914848 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:12.914834    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-cgd7g"
Sep 09 04:02:12.917249 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:12.917228    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 04:02:12.917941 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:12.917912    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-prv5d\""
Sep 09 04:02:12.918061 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:12.917957    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 04:02:12.922576 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:12.922556    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxfqj\" (UniqueName: \"kubernetes.io/projected/6df5a24f-bc92-457b-ac81-d585aa2c5aca-kube-api-access-rxfqj\") pod \"cert-manager-webhook-6486ffd997-cgd7g\" (UID: \"6df5a24f-bc92-457b-ac81-d585aa2c5aca\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cgd7g"
Sep 09 04:02:12.922701 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:12.922603    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6df5a24f-bc92-457b-ac81-d585aa2c5aca-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cgd7g\" (UID: \"6df5a24f-bc92-457b-ac81-d585aa2c5aca\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cgd7g"
Sep 09 04:02:12.922882 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:12.922864    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cgd7g"]
Sep 09 04:02:13.023158 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:13.023133    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxfqj\" (UniqueName: \"kubernetes.io/projected/6df5a24f-bc92-457b-ac81-d585aa2c5aca-kube-api-access-rxfqj\") pod \"cert-manager-webhook-6486ffd997-cgd7g\" (UID: \"6df5a24f-bc92-457b-ac81-d585aa2c5aca\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cgd7g"
Sep 09 04:02:13.023284 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:13.023188    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6df5a24f-bc92-457b-ac81-d585aa2c5aca-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cgd7g\" (UID: \"6df5a24f-bc92-457b-ac81-d585aa2c5aca\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cgd7g"
Sep 09 04:02:13.030986 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:13.030961    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6df5a24f-bc92-457b-ac81-d585aa2c5aca-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cgd7g\" (UID: \"6df5a24f-bc92-457b-ac81-d585aa2c5aca\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cgd7g"
Sep 09 04:02:13.031088 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:13.031012    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxfqj\" (UniqueName: \"kubernetes.io/projected/6df5a24f-bc92-457b-ac81-d585aa2c5aca-kube-api-access-rxfqj\") pod \"cert-manager-webhook-6486ffd997-cgd7g\" (UID: \"6df5a24f-bc92-457b-ac81-d585aa2c5aca\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cgd7g"
Sep 09 04:02:13.238417 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:13.238398    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-cgd7g"
Sep 09 04:02:13.506391 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:13.506322    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cgd7g"]
Sep 09 04:02:13.509281 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:02:13.509255    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6df5a24f_bc92_457b_ac81_d585aa2c5aca.slice/crio-ab6fba618e8ec81a398a928537c9953b71c42c2f564a730ae686e387b92ddb4f WatchSource:0}: Error finding container ab6fba618e8ec81a398a928537c9953b71c42c2f564a730ae686e387b92ddb4f: Status 404 returned error can't find the container with id ab6fba618e8ec81a398a928537c9953b71c42c2f564a730ae686e387b92ddb4f
Sep 09 04:02:14.150645 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:14.150605    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-cgd7g" event={"ID":"6df5a24f-bc92-457b-ac81-d585aa2c5aca","Type":"ContainerStarted","Data":"ab6fba618e8ec81a398a928537c9953b71c42c2f564a730ae686e387b92ddb4f"}
Sep 09 04:02:16.749635 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:16.749604    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:16.808361 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:16.808289    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:17.166172 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:17.166085    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-cgd7g" event={"ID":"6df5a24f-bc92-457b-ac81-d585aa2c5aca","Type":"ContainerStarted","Data":"5c39b6c691e35a425daf74ebf0391821085a61da934eaf1d264e74061841fd8c"}
Sep 09 04:02:17.166334 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:17.166208    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-cgd7g"
Sep 09 04:02:17.182394 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:17.182348    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-cgd7g" podStartSLOduration=2.163939384 podStartE2EDuration="5.182330175s" totalImagesPullingTime="3.018390791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:02:17.181203485 +0000 UTC m=+236.707574194" watchObservedRunningTime="2026-09-09 04:02:17.182330175 +0000 UTC m=+236.708700886"
Sep 09 04:02:18.992075 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:18.992042    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:19.092636 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:19.092581    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:23.171591 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:23.171559    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-cgd7g"
Sep 09 04:02:24.606031 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:24.605993    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:24.706661 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:24.706630    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:26.387238 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:26.387202    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:42.744623 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:42.744541    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:42.842301 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:42.842270    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:43.355978 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.355948    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:43.619619 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.619513    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:43.722652 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.722624    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:43.774306 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.774280    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"]
Sep 09 04:02:43.777685 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.777669    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.779999 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.779978    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 04:02:43.780114 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.780016    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 04:02:43.780179 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.780116    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 04:02:43.780368 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.780353    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 04:02:43.780466 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.780435    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2jwxl\""
Sep 09 04:02:43.787378 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.787360    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"]
Sep 09 04:02:43.857725 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.857704    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88p68\" (UniqueName: \"kubernetes.io/projected/7cbb60ae-799c-4198-8e94-c5aed9cff219-kube-api-access-88p68\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.857835 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.857733    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7cbb60ae-799c-4198-8e94-c5aed9cff219-manifests\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.857835 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.857752    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7cbb60ae-799c-4198-8e94-c5aed9cff219-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.857835 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.857771    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cbb60ae-799c-4198-8e94-c5aed9cff219-tmp\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.857835 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.857796    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7cbb60ae-799c-4198-8e94-c5aed9cff219-webhook-cert\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.870535 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.870482    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:43.958197 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.958175    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88p68\" (UniqueName: \"kubernetes.io/projected/7cbb60ae-799c-4198-8e94-c5aed9cff219-kube-api-access-88p68\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.958297 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.958206    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7cbb60ae-799c-4198-8e94-c5aed9cff219-manifests\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.958297 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.958228    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7cbb60ae-799c-4198-8e94-c5aed9cff219-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.958297 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.958258    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cbb60ae-799c-4198-8e94-c5aed9cff219-tmp\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.958297 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.958288    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7cbb60ae-799c-4198-8e94-c5aed9cff219-webhook-cert\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.958653 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.958633    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7cbb60ae-799c-4198-8e94-c5aed9cff219-manifests\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.960696 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.960673    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cbb60ae-799c-4198-8e94-c5aed9cff219-tmp\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.960753 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.960699    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7cbb60ae-799c-4198-8e94-c5aed9cff219-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.960884 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.960870    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7cbb60ae-799c-4198-8e94-c5aed9cff219-webhook-cert\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:43.966198 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:43.966173    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88p68\" (UniqueName: \"kubernetes.io/projected/7cbb60ae-799c-4198-8e94-c5aed9cff219-kube-api-access-88p68\") pod \"opendatahub-operator-controller-manager-5996d6cf95-nx2mg\" (UID: \"7cbb60ae-799c-4198-8e94-c5aed9cff219\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:44.088587 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:44.088565    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:44.224765 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:44.224738    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:44.324373 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:44.324342    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:44.365620 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:44.365601    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"]
Sep 09 04:02:44.368028 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:02:44.368001    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbb60ae_799c_4198_8e94_c5aed9cff219.slice/crio-fbf4300706cece18892656cd9500048ca820ec7e0a8a4fb9464feb134554a5a8 WatchSource:0}: Error finding container fbf4300706cece18892656cd9500048ca820ec7e0a8a4fb9464feb134554a5a8: Status 404 returned error can't find the container with id fbf4300706cece18892656cd9500048ca820ec7e0a8a4fb9464feb134554a5a8
Sep 09 04:02:44.425369 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:44.425299    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:44.529654 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:44.529629    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:45.279119 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:45.279079    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg" event={"ID":"7cbb60ae-799c-4198-8e94-c5aed9cff219","Type":"ContainerStarted","Data":"fbf4300706cece18892656cd9500048ca820ec7e0a8a4fb9464feb134554a5a8"}
Sep 09 04:02:46.703888 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:46.703860    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:46.805757 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:46.805719    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:48.295127 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:48.295093    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg" event={"ID":"7cbb60ae-799c-4198-8e94-c5aed9cff219","Type":"ContainerStarted","Data":"901bb007acbf1806ab18084157e0bb847dcf5a6d630a74d9772f8128c12fa308"}
Sep 09 04:02:49.301210 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:49.301180    2553 generic.go:356] "Generic (PLEG): container finished" podID="7cbb60ae-799c-4198-8e94-c5aed9cff219" containerID="901bb007acbf1806ab18084157e0bb847dcf5a6d630a74d9772f8128c12fa308" exitCode=0
Sep 09 04:02:49.301637 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:49.301250    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg" event={"ID":"7cbb60ae-799c-4198-8e94-c5aed9cff219","Type":"ContainerDied","Data":"901bb007acbf1806ab18084157e0bb847dcf5a6d630a74d9772f8128c12fa308"}
Sep 09 04:02:50.306961 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:50.306919    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg" event={"ID":"7cbb60ae-799c-4198-8e94-c5aed9cff219","Type":"ContainerStarted","Data":"c8c6ac25853c0608c71d2cce29c96978ce62782c88f5ae02830e6af38a9303ff"}
Sep 09 04:02:50.307305 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:50.307007    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:02:50.327278 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:50.327239    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg" podStartSLOduration=2.692889049 podStartE2EDuration="7.327226449s" totalImagesPullingTime="3.617062796s" totalInitContainerRuntime="1.017274604s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:02:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:02:50.326253126 +0000 UTC m=+269.852623837" watchObservedRunningTime="2026-09-09 04:02:50.327226449 +0000 UTC m=+269.853597156"
Sep 09 04:02:55.760141 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:55.760098    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:02:55.865751 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:02:55.865717    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:01.313257 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:01.313234    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-nx2mg"
Sep 09 04:03:02.801998 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:02.801950    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:03.171717 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.171647    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj"]
Sep 09 04:03:03.180027 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.180008    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj"
Sep 09 04:03:03.183559 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.183535    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gwzpt\""
Sep 09 04:03:03.184218 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.184060    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 04:03:03.184218 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.184069    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 04:03:03.184218 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.184118    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 04:03:03.184653 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.184625    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 04:03:03.186066 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.186046    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj"]
Sep 09 04:03:03.311917 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.311889    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/822e9f2c-cd73-42f1-9f9f-39f752bea97e-tmp\") pod \"kube-auth-proxy-7449ffbbd4-g5hqj\" (UID: \"822e9f2c-cd73-42f1-9f9f-39f752bea97e\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj"
Sep 09 04:03:03.312101 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.311925    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/822e9f2c-cd73-42f1-9f9f-39f752bea97e-tls-certs\") pod \"kube-auth-proxy-7449ffbbd4-g5hqj\" (UID: \"822e9f2c-cd73-42f1-9f9f-39f752bea97e\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj"
Sep 09 04:03:03.312101 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.311949    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cz6l\" (UniqueName: \"kubernetes.io/projected/822e9f2c-cd73-42f1-9f9f-39f752bea97e-kube-api-access-4cz6l\") pod \"kube-auth-proxy-7449ffbbd4-g5hqj\" (UID: \"822e9f2c-cd73-42f1-9f9f-39f752bea97e\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj"
Sep 09 04:03:03.412562 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.412537    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/822e9f2c-cd73-42f1-9f9f-39f752bea97e-tls-certs\") pod \"kube-auth-proxy-7449ffbbd4-g5hqj\" (UID: \"822e9f2c-cd73-42f1-9f9f-39f752bea97e\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj"
Sep 09 04:03:03.412677 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.412572    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cz6l\" (UniqueName: \"kubernetes.io/projected/822e9f2c-cd73-42f1-9f9f-39f752bea97e-kube-api-access-4cz6l\") pod \"kube-auth-proxy-7449ffbbd4-g5hqj\" (UID: \"822e9f2c-cd73-42f1-9f9f-39f752bea97e\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj"
Sep 09 04:03:03.412677 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.412638    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/822e9f2c-cd73-42f1-9f9f-39f752bea97e-tmp\") pod \"kube-auth-proxy-7449ffbbd4-g5hqj\" (UID: \"822e9f2c-cd73-42f1-9f9f-39f752bea97e\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj"
Sep 09 04:03:03.414973 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.414948    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/822e9f2c-cd73-42f1-9f9f-39f752bea97e-tmp\") pod \"kube-auth-proxy-7449ffbbd4-g5hqj\" (UID: \"822e9f2c-cd73-42f1-9f9f-39f752bea97e\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj"
Sep 09 04:03:03.415095 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.415078    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/822e9f2c-cd73-42f1-9f9f-39f752bea97e-tls-certs\") pod \"kube-auth-proxy-7449ffbbd4-g5hqj\" (UID: \"822e9f2c-cd73-42f1-9f9f-39f752bea97e\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj"
Sep 09 04:03:03.420651 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.420629    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cz6l\" (UniqueName: \"kubernetes.io/projected/822e9f2c-cd73-42f1-9f9f-39f752bea97e-kube-api-access-4cz6l\") pod \"kube-auth-proxy-7449ffbbd4-g5hqj\" (UID: \"822e9f2c-cd73-42f1-9f9f-39f752bea97e\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj"
Sep 09 04:03:03.491508 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.491485    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj"
Sep 09 04:03:03.784574 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:03.784554    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj"]
Sep 09 04:03:03.786817 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:03:03.786790    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822e9f2c_cd73_42f1_9f9f_39f752bea97e.slice/crio-2bc07685d5525f5b981c18754201666b10573c904363b187097e0ccfc52a6a33 WatchSource:0}: Error finding container 2bc07685d5525f5b981c18754201666b10573c904363b187097e0ccfc52a6a33: Status 404 returned error can't find the container with id 2bc07685d5525f5b981c18754201666b10573c904363b187097e0ccfc52a6a33
Sep 09 04:03:04.365340 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:04.365308    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj" event={"ID":"822e9f2c-cd73-42f1-9f9f-39f752bea97e","Type":"ContainerStarted","Data":"2bc07685d5525f5b981c18754201666b10573c904363b187097e0ccfc52a6a33"}
Sep 09 04:03:07.386275 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:07.386242    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj" event={"ID":"822e9f2c-cd73-42f1-9f9f-39f752bea97e","Type":"ContainerStarted","Data":"fdd0d03ac95293c5c25d4fa7609b87be9d2511b870eacf6308a5e301b0687c82"}
Sep 09 04:03:07.404138 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:07.404094    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-g5hqj" podStartSLOduration=1.35891107 podStartE2EDuration="4.404080707s" totalImagesPullingTime="3.045169637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:03:07.402504675 +0000 UTC m=+286.928875386" watchObservedRunningTime="2026-09-09 04:03:07.404080707 +0000 UTC m=+286.930451416"
Sep 09 04:03:07.885903 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:07.885877    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:09.730039 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:09.730002    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:09.833155 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:09.833118    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:10.086167 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.086088    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"]
Sep 09 04:03:10.089600 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.089576    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.091677 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.091650    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 04:03:10.091925 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.091909    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7gpbd\""
Sep 09 04:03:10.100591 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.100572    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"]
Sep 09 04:03:10.122985 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.122944    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:10.170818 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.170794    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.170952 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.170828    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrxbk\" (UniqueName: \"kubernetes.io/projected/1eea3e85-da66-419b-9e8c-890e5a6afc6d-kube-api-access-nrxbk\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.170952 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.170860    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.170952 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.170914    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.171081 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.170951    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.171081 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.171016    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.171081 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.171055    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.171081 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.171072    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.171226 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.171087    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.237388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.237371    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:10.271535 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.271517    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.271633 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.271566    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.271633 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.271592    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.271633 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.271606    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.271633 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.271621    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.271837 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.271710    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.271837 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.271753    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrxbk\" (UniqueName: \"kubernetes.io/projected/1eea3e85-da66-419b-9e8c-890e5a6afc6d-kube-api-access-nrxbk\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.271837 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.271815    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.271983 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.271845    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.272062 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.272038    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.272152 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.272130    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.272217 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.272184    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.272269 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.272233    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.272269 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.272249    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.274118 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.274099    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.274436 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.274417    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.279668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.279651    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrxbk\" (UniqueName: \"kubernetes.io/projected/1eea3e85-da66-419b-9e8c-890e5a6afc6d-kube-api-access-nrxbk\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.279753 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.279684    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eea3e85-da66-419b-9e8c-890e5a6afc6d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2\" (UID: \"1eea3e85-da66-419b-9e8c-890e5a6afc6d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.401956 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.401891    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:10.437308 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.437270    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:10.675076 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:10.675053    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"]
Sep 09 04:03:10.676725 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:03:10.676699    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eea3e85_da66_419b_9e8c_890e5a6afc6d.slice/crio-bdac46d6c4ae38fa93677144add2f638078c267f07fa5eeca09b23645e717e2a WatchSource:0}: Error finding container bdac46d6c4ae38fa93677144add2f638078c267f07fa5eeca09b23645e717e2a: Status 404 returned error can't find the container with id bdac46d6c4ae38fa93677144add2f638078c267f07fa5eeca09b23645e717e2a
Sep 09 04:03:11.403652 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:11.403617    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2" event={"ID":"1eea3e85-da66-419b-9e8c-890e5a6afc6d","Type":"ContainerStarted","Data":"bdac46d6c4ae38fa93677144add2f638078c267f07fa5eeca09b23645e717e2a"}
Sep 09 04:03:13.537665 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:13.537628    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:03:13.537923 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:13.537708    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:03:13.537923 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:13.537737    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:03:14.120841 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.120792    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:14.220458 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.220422    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:14.320287 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.320258    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:14.421341 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.421224    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2" event={"ID":"1eea3e85-da66-419b-9e8c-890e5a6afc6d","Type":"ContainerStarted","Data":"afc98c15d9dda0207ee81a887d28283fea66a6d9538e6ddd78c3482863ebbd6c"}
Sep 09 04:03:14.421675 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.421652    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:14.440457 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.440390    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2" podStartSLOduration=1.581787946 podStartE2EDuration="4.440376127s" totalImagesPullingTime="2.858588181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:03:14.439404003 +0000 UTC m=+293.965774711" watchObservedRunningTime="2026-09-09 04:03:14.440376127 +0000 UTC m=+293.966746836"
Sep 09 04:03:14.718068 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.717995    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pq5qh"]
Sep 09 04:03:14.721420 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.721402    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pq5qh"
Sep 09 04:03:14.723628 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.723597    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 09 04:03:14.723731 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.723601    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-bl84t\""
Sep 09 04:03:14.731353 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.731328    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pq5qh"]
Sep 09 04:03:14.807831 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.807807    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxd28\" (UniqueName: \"kubernetes.io/projected/f2741325-8004-410d-9682-26b1e1f8b6cb-kube-api-access-pxd28\") pod \"model-serving-api-54d89bd5fc-pq5qh\" (UID: \"f2741325-8004-410d-9682-26b1e1f8b6cb\") " pod="opendatahub/model-serving-api-54d89bd5fc-pq5qh"
Sep 09 04:03:14.807980 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.807897    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2741325-8004-410d-9682-26b1e1f8b6cb-tls-certs\") pod \"model-serving-api-54d89bd5fc-pq5qh\" (UID: \"f2741325-8004-410d-9682-26b1e1f8b6cb\") " pod="opendatahub/model-serving-api-54d89bd5fc-pq5qh"
Sep 09 04:03:14.909128 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.909104    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxd28\" (UniqueName: \"kubernetes.io/projected/f2741325-8004-410d-9682-26b1e1f8b6cb-kube-api-access-pxd28\") pod \"model-serving-api-54d89bd5fc-pq5qh\" (UID: \"f2741325-8004-410d-9682-26b1e1f8b6cb\") " pod="opendatahub/model-serving-api-54d89bd5fc-pq5qh"
Sep 09 04:03:14.909240 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.909168    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2741325-8004-410d-9682-26b1e1f8b6cb-tls-certs\") pod \"model-serving-api-54d89bd5fc-pq5qh\" (UID: \"f2741325-8004-410d-9682-26b1e1f8b6cb\") " pod="opendatahub/model-serving-api-54d89bd5fc-pq5qh"
Sep 09 04:03:14.909295 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:03:14.909260    2553 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 09 04:03:14.909359 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:03:14.909343    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2741325-8004-410d-9682-26b1e1f8b6cb-tls-certs podName:f2741325-8004-410d-9682-26b1e1f8b6cb nodeName:}" failed. No retries permitted until 2026-09-09 04:03:15.409307381 +0000 UTC m=+294.935678070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f2741325-8004-410d-9682-26b1e1f8b6cb-tls-certs") pod "model-serving-api-54d89bd5fc-pq5qh" (UID: "f2741325-8004-410d-9682-26b1e1f8b6cb") : secret "model-serving-api-tls" not found
Sep 09 04:03:14.917671 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:14.917649    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxd28\" (UniqueName: \"kubernetes.io/projected/f2741325-8004-410d-9682-26b1e1f8b6cb-kube-api-access-pxd28\") pod \"model-serving-api-54d89bd5fc-pq5qh\" (UID: \"f2741325-8004-410d-9682-26b1e1f8b6cb\") " pod="opendatahub/model-serving-api-54d89bd5fc-pq5qh"
Sep 09 04:03:15.402480 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:15.402432    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:15.406834 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:15.406811    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:15.414389 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:15.414366    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2741325-8004-410d-9682-26b1e1f8b6cb-tls-certs\") pod \"model-serving-api-54d89bd5fc-pq5qh\" (UID: \"f2741325-8004-410d-9682-26b1e1f8b6cb\") " pod="opendatahub/model-serving-api-54d89bd5fc-pq5qh"
Sep 09 04:03:15.416779 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:15.416756    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2741325-8004-410d-9682-26b1e1f8b6cb-tls-certs\") pod \"model-serving-api-54d89bd5fc-pq5qh\" (UID: \"f2741325-8004-410d-9682-26b1e1f8b6cb\") " pod="opendatahub/model-serving-api-54d89bd5fc-pq5qh"
Sep 09 04:03:15.426485 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:15.426434    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:15.427353 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:15.427336    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2"
Sep 09 04:03:15.632915 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:15.632886    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pq5qh"
Sep 09 04:03:15.757479 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:15.757424    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:15.806859 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:15.806833    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:15.902162 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:15.902136    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pq5qh"]
Sep 09 04:03:15.904089 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:03:15.904050    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2741325_8004_410d_9682_26b1e1f8b6cb.slice/crio-b9a146c0f7a607298e017f65903ae441c2a3c253a900b3616a4ed3d0ac620466 WatchSource:0}: Error finding container b9a146c0f7a607298e017f65903ae441c2a3c253a900b3616a4ed3d0ac620466: Status 404 returned error can't find the container with id b9a146c0f7a607298e017f65903ae441c2a3c253a900b3616a4ed3d0ac620466
Sep 09 04:03:16.431348 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:16.431317    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pq5qh" event={"ID":"f2741325-8004-410d-9682-26b1e1f8b6cb","Type":"ContainerStarted","Data":"b9a146c0f7a607298e017f65903ae441c2a3c253a900b3616a4ed3d0ac620466"}
Sep 09 04:03:18.443848 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:18.443812    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pq5qh" event={"ID":"f2741325-8004-410d-9682-26b1e1f8b6cb","Type":"ContainerStarted","Data":"960345bbc9f2e0d923badb43c45e6398716611a886b53f1196c27a1cb01e0bb2"}
Sep 09 04:03:18.444278 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:18.443908    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-pq5qh"
Sep 09 04:03:18.462000 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:18.461956    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-pq5qh" podStartSLOduration=2.899987223 podStartE2EDuration="4.461940644s" totalImagesPullingTime="1.561953421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:03:18.46069381 +0000 UTC m=+297.987064529" watchObservedRunningTime="2026-09-09 04:03:18.461940644 +0000 UTC m=+297.988311399"
Sep 09 04:03:20.873332 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:20.873307    2553 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 04:03:23.070335 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.070301    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"]
Sep 09 04:03:23.076060 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.076044    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:23.079927 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.079907    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 04:03:23.080080 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.079980    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kd6lp\""
Sep 09 04:03:23.080080 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.080052    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 04:03:23.080080 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.080065    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 04:03:23.080223 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.080173    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 04:03:23.080275 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.080233    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 04:03:23.084226 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.084208    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"]
Sep 09 04:03:23.179384 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.179357    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/74b48f4d-caa8-40d5-9f9d-0a60e4d8651e-manager-config\") pod \"lws-controller-manager-7486b797d6-b6ggb\" (UID: \"74b48f4d-caa8-40d5-9f9d-0a60e4d8651e\") " pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:23.179384 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.179390    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/74b48f4d-caa8-40d5-9f9d-0a60e4d8651e-metrics-cert\") pod \"lws-controller-manager-7486b797d6-b6ggb\" (UID: \"74b48f4d-caa8-40d5-9f9d-0a60e4d8651e\") " pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:23.179591 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.179531    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77qdv\" (UniqueName: \"kubernetes.io/projected/74b48f4d-caa8-40d5-9f9d-0a60e4d8651e-kube-api-access-77qdv\") pod \"lws-controller-manager-7486b797d6-b6ggb\" (UID: \"74b48f4d-caa8-40d5-9f9d-0a60e4d8651e\") " pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:23.179627 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.179615    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74b48f4d-caa8-40d5-9f9d-0a60e4d8651e-cert\") pod \"lws-controller-manager-7486b797d6-b6ggb\" (UID: \"74b48f4d-caa8-40d5-9f9d-0a60e4d8651e\") " pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:23.280782 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.280747    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77qdv\" (UniqueName: \"kubernetes.io/projected/74b48f4d-caa8-40d5-9f9d-0a60e4d8651e-kube-api-access-77qdv\") pod \"lws-controller-manager-7486b797d6-b6ggb\" (UID: \"74b48f4d-caa8-40d5-9f9d-0a60e4d8651e\") " pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:23.280973 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.280809    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74b48f4d-caa8-40d5-9f9d-0a60e4d8651e-cert\") pod \"lws-controller-manager-7486b797d6-b6ggb\" (UID: \"74b48f4d-caa8-40d5-9f9d-0a60e4d8651e\") " pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:23.280973 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.280856    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/74b48f4d-caa8-40d5-9f9d-0a60e4d8651e-manager-config\") pod \"lws-controller-manager-7486b797d6-b6ggb\" (UID: \"74b48f4d-caa8-40d5-9f9d-0a60e4d8651e\") " pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:23.280973 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.280882    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/74b48f4d-caa8-40d5-9f9d-0a60e4d8651e-metrics-cert\") pod \"lws-controller-manager-7486b797d6-b6ggb\" (UID: \"74b48f4d-caa8-40d5-9f9d-0a60e4d8651e\") " pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:23.281636 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.281604    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/74b48f4d-caa8-40d5-9f9d-0a60e4d8651e-manager-config\") pod \"lws-controller-manager-7486b797d6-b6ggb\" (UID: \"74b48f4d-caa8-40d5-9f9d-0a60e4d8651e\") " pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:23.283593 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.283572    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74b48f4d-caa8-40d5-9f9d-0a60e4d8651e-cert\") pod \"lws-controller-manager-7486b797d6-b6ggb\" (UID: \"74b48f4d-caa8-40d5-9f9d-0a60e4d8651e\") " pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:23.283735 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.283716    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/74b48f4d-caa8-40d5-9f9d-0a60e4d8651e-metrics-cert\") pod \"lws-controller-manager-7486b797d6-b6ggb\" (UID: \"74b48f4d-caa8-40d5-9f9d-0a60e4d8651e\") " pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:23.293965 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.293940    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77qdv\" (UniqueName: \"kubernetes.io/projected/74b48f4d-caa8-40d5-9f9d-0a60e4d8651e-kube-api-access-77qdv\") pod \"lws-controller-manager-7486b797d6-b6ggb\" (UID: \"74b48f4d-caa8-40d5-9f9d-0a60e4d8651e\") " pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:23.385868 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.385808    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:23.672086 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.672062    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"]
Sep 09 04:03:23.674503 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:03:23.674477    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b48f4d_caa8_40d5_9f9d_0a60e4d8651e.slice/crio-22ef70e33bc21a8813cefc7d6ed4bd9f2a72a1941eef6eff0582d5b9ffdcd20f WatchSource:0}: Error finding container 22ef70e33bc21a8813cefc7d6ed4bd9f2a72a1941eef6eff0582d5b9ffdcd20f: Status 404 returned error can't find the container with id 22ef70e33bc21a8813cefc7d6ed4bd9f2a72a1941eef6eff0582d5b9ffdcd20f
Sep 09 04:03:23.676437 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:23.676422    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 04:03:24.318633 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:24.318597    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:24.390572 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:24.390532    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:24.470493 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:24.470438    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb" event={"ID":"74b48f4d-caa8-40d5-9f9d-0a60e4d8651e","Type":"ContainerStarted","Data":"22ef70e33bc21a8813cefc7d6ed4bd9f2a72a1941eef6eff0582d5b9ffdcd20f"}
Sep 09 04:03:24.571133 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:24.571052    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:24.672508 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:24.672471    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:25.604706 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:25.604675    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:25.700346 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:25.700311    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:25.805308 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:25.805264    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:03:26.482046 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:26.482013    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb" event={"ID":"74b48f4d-caa8-40d5-9f9d-0a60e4d8651e","Type":"ContainerStarted","Data":"e4eebdf8a3c56a081bb5009f28072e82053f01882c4addff1d519b5127ddd2a5"}
Sep 09 04:03:26.482220 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:26.482135    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:26.500706 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:26.500660    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb" podStartSLOduration=1.006415769 podStartE2EDuration="3.500645825s" totalImagesPullingTime="2.494230056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:03:26.498142847 +0000 UTC m=+306.024513572" watchObservedRunningTime="2026-09-09 04:03:26.500645825 +0000 UTC m=+306.027016534"
Sep 09 04:03:29.452450 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:29.452423    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-pq5qh"
Sep 09 04:03:37.489223 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:37.489183    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7486b797d6-b6ggb"
Sep 09 04:03:49.271824 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:03:49.271795    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:15.074721 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:15.074648    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:15.128101 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:15.128057    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:23.140405 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:23.140372    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:26.789692 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:26.789661    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:27.580630 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:27.580596    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:30.657628 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:30.657595    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:30.800457 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:30.800398    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:31.850600 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:31.850561    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:32.630083 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:32.630050    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:32.794040 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:32.793995    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:33.403545 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:33.403512    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:33.635909 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:33.635877    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:33.732668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:33.732639    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:33.833554 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:33.833524    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:33.938061 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:33.938034    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:37.009058 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:37.009026    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:38.025549 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:38.025514    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:40.010412 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:40.010357    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:40.066776 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:40.066751    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:40.394395 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:40.394304    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:40.494579 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:40.494541    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:42.708632 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:42.708590    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:42.757758 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:42.757721    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:42.814245 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:42.814219    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:43.820101 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:43.820067    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:43.930968 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:43.930942    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:44.069257 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:44.069224    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:44.170355 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:44.170293    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:48.350472 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:48.350426    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:49.579678 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:49.579649    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:49.635966 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:49.635939    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:51.763475 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:51.763416    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:51.819269 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:51.819225    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:53.620779 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:53.620728    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:53.720791 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:53.720761    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:54.602646 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.602614    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"]
Sep 09 04:04:54.605890 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.605872    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"
Sep 09 04:04:54.608319 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.608297    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 04:04:54.608433 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.608350    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vg2bx\""
Sep 09 04:04:54.608662 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.608647    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 04:04:54.620663 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.620641    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"]
Sep 09 04:04:54.712608 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.712580    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltx6l\" (UniqueName: \"kubernetes.io/projected/3f4f4a75-c3e1-43d5-8f7f-769937f40e11-kube-api-access-ltx6l\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-mf99j\" (UID: \"3f4f4a75-c3e1-43d5-8f7f-769937f40e11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"
Sep 09 04:04:54.713023 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.712673    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f4f4a75-c3e1-43d5-8f7f-769937f40e11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-mf99j\" (UID: \"3f4f4a75-c3e1-43d5-8f7f-769937f40e11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"
Sep 09 04:04:54.813345 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.813297    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltx6l\" (UniqueName: \"kubernetes.io/projected/3f4f4a75-c3e1-43d5-8f7f-769937f40e11-kube-api-access-ltx6l\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-mf99j\" (UID: \"3f4f4a75-c3e1-43d5-8f7f-769937f40e11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"
Sep 09 04:04:54.813531 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.813370    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f4f4a75-c3e1-43d5-8f7f-769937f40e11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-mf99j\" (UID: \"3f4f4a75-c3e1-43d5-8f7f-769937f40e11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"
Sep 09 04:04:54.813717 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.813699    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f4f4a75-c3e1-43d5-8f7f-769937f40e11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-mf99j\" (UID: \"3f4f4a75-c3e1-43d5-8f7f-769937f40e11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"
Sep 09 04:04:54.832390 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.832367    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltx6l\" (UniqueName: \"kubernetes.io/projected/3f4f4a75-c3e1-43d5-8f7f-769937f40e11-kube-api-access-ltx6l\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-mf99j\" (UID: \"3f4f4a75-c3e1-43d5-8f7f-769937f40e11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"
Sep 09 04:04:54.864820 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.864772    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:54.916617 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.916592    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"
Sep 09 04:04:54.968589 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:54.968560    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:04:55.192526 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:55.192502    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"]
Sep 09 04:04:55.194646 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:04:55.194620    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4f4a75_c3e1_43d5_8f7f_769937f40e11.slice/crio-c17bdf7eed4b78906ca2786901b5186b53c4c14abd2f2e3dff4a544565dbe10e WatchSource:0}: Error finding container c17bdf7eed4b78906ca2786901b5186b53c4c14abd2f2e3dff4a544565dbe10e: Status 404 returned error can't find the container with id c17bdf7eed4b78906ca2786901b5186b53c4c14abd2f2e3dff4a544565dbe10e
Sep 09 04:04:55.877617 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:55.877584    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j" event={"ID":"3f4f4a75-c3e1-43d5-8f7f-769937f40e11","Type":"ContainerStarted","Data":"c17bdf7eed4b78906ca2786901b5186b53c4c14abd2f2e3dff4a544565dbe10e"}
Sep 09 04:04:59.901937 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:59.901908    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j" event={"ID":"3f4f4a75-c3e1-43d5-8f7f-769937f40e11","Type":"ContainerStarted","Data":"7105fc8f6bcac48fe7ab732b512dafc1770112afe49ec78c3f146badc585945c"}
Sep 09 04:04:59.902302 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:59.902037    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"
Sep 09 04:04:59.923839 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:04:59.923783    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j" podStartSLOduration=2.004471624 podStartE2EDuration="5.923766844s" totalImagesPullingTime="3.91929522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:04:59.92143255 +0000 UTC m=+399.447803254" watchObservedRunningTime="2026-09-09 04:04:59.923766844 +0000 UTC m=+399.450137554"
Sep 09 04:05:00.383818 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:00.383790    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:00.437666 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:00.437635    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:00.977004 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:00.976973    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:01.176283 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:01.176252    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:04.376771 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:04.376738    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:04.477224 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:04.477195    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:08.744368 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:08.744334    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:08.844419 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:08.844395    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:08.987532 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:08.987504    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:09.058500 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:09.058416    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:10.908939 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:10.908911    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"
Sep 09 04:05:14.918670 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:14.918638    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:15.008694 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:15.008665    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:24.799546 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:24.799516    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"]
Sep 09 04:05:24.799995 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:24.799728    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j" podUID="3f4f4a75-c3e1-43d5-8f7f-769937f40e11" containerName="manager" containerID="cri-o://7105fc8f6bcac48fe7ab732b512dafc1770112afe49ec78c3f146badc585945c" gracePeriod=10
Sep 09 04:05:25.023397 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:25.023364    2553 generic.go:356] "Generic (PLEG): container finished" podID="3f4f4a75-c3e1-43d5-8f7f-769937f40e11" containerID="7105fc8f6bcac48fe7ab732b512dafc1770112afe49ec78c3f146badc585945c" exitCode=0
Sep 09 04:05:25.023574 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:25.023432    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j" event={"ID":"3f4f4a75-c3e1-43d5-8f7f-769937f40e11","Type":"ContainerDied","Data":"7105fc8f6bcac48fe7ab732b512dafc1770112afe49ec78c3f146badc585945c"}
Sep 09 04:05:25.042535 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:25.042517    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"
Sep 09 04:05:25.145958 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:25.145889    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltx6l\" (UniqueName: \"kubernetes.io/projected/3f4f4a75-c3e1-43d5-8f7f-769937f40e11-kube-api-access-ltx6l\") pod \"3f4f4a75-c3e1-43d5-8f7f-769937f40e11\" (UID: \"3f4f4a75-c3e1-43d5-8f7f-769937f40e11\") "
Sep 09 04:05:25.145958 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:25.145953    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f4f4a75-c3e1-43d5-8f7f-769937f40e11-extensions-socket-volume\") pod \"3f4f4a75-c3e1-43d5-8f7f-769937f40e11\" (UID: \"3f4f4a75-c3e1-43d5-8f7f-769937f40e11\") "
Sep 09 04:05:25.146292 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:25.146270    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4f4a75-c3e1-43d5-8f7f-769937f40e11-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3f4f4a75-c3e1-43d5-8f7f-769937f40e11" (UID: "3f4f4a75-c3e1-43d5-8f7f-769937f40e11"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:05:25.148087 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:25.148068    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4f4a75-c3e1-43d5-8f7f-769937f40e11-kube-api-access-ltx6l" (OuterVolumeSpecName: "kube-api-access-ltx6l") pod "3f4f4a75-c3e1-43d5-8f7f-769937f40e11" (UID: "3f4f4a75-c3e1-43d5-8f7f-769937f40e11"). InnerVolumeSpecName "kube-api-access-ltx6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:05:25.247021 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:25.246995    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltx6l\" (UniqueName: \"kubernetes.io/projected/3f4f4a75-c3e1-43d5-8f7f-769937f40e11-kube-api-access-ltx6l\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:05:25.247021 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:25.247017    2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f4f4a75-c3e1-43d5-8f7f-769937f40e11-extensions-socket-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:05:26.029727 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:26.029704    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"
Sep 09 04:05:26.030203 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:26.029704    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j" event={"ID":"3f4f4a75-c3e1-43d5-8f7f-769937f40e11","Type":"ContainerDied","Data":"c17bdf7eed4b78906ca2786901b5186b53c4c14abd2f2e3dff4a544565dbe10e"}
Sep 09 04:05:26.030203 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:26.029814    2553 scope.go:117] "RemoveContainer" containerID="7105fc8f6bcac48fe7ab732b512dafc1770112afe49ec78c3f146badc585945c"
Sep 09 04:05:26.055962 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:26.055941    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"]
Sep 09 04:05:26.059596 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:26.059576    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-mf99j"]
Sep 09 04:05:26.988283 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:26.988241    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f4f4a75-c3e1-43d5-8f7f-769937f40e11" path="/var/lib/kubelet/pods/3f4f4a75-c3e1-43d5-8f7f-769937f40e11/volumes"
Sep 09 04:05:27.275832 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.275758    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"]
Sep 09 04:05:27.276153 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.276134    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f4f4a75-c3e1-43d5-8f7f-769937f40e11" containerName="manager"
Sep 09 04:05:27.276153 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.276145    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f4f4a75-c3e1-43d5-8f7f-769937f40e11" containerName="manager"
Sep 09 04:05:27.276222 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.276196    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f4f4a75-c3e1-43d5-8f7f-769937f40e11" containerName="manager"
Sep 09 04:05:27.280571 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.280553    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.282808 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.282789    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rb44t\""
Sep 09 04:05:27.291198 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.291178    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"]
Sep 09 04:05:27.299466 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.299425    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:27.362135 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.362114    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f77855-b526-401c-b8a5-3ccf9360997c-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.362250 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.362157    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.362250 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.362183    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.362377 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.362255    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.362377 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.362296    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.362377 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.362336    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5266s\" (UniqueName: \"kubernetes.io/projected/f8f77855-b526-401c-b8a5-3ccf9360997c-kube-api-access-5266s\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.362571 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.362412    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f77855-b526-401c-b8a5-3ccf9360997c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.362571 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.362472    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.362571 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.362509    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f77855-b526-401c-b8a5-3ccf9360997c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.463284 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.463256    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f77855-b526-401c-b8a5-3ccf9360997c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.463492 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.463293    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.463492 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.463325    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f77855-b526-401c-b8a5-3ccf9360997c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.463492 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.463364    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f77855-b526-401c-b8a5-3ccf9360997c-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.463492 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.463415    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.463492 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.463469    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.464106 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.463503    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.464106 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.463533    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.464106 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.463566    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5266s\" (UniqueName: \"kubernetes.io/projected/f8f77855-b526-401c-b8a5-3ccf9360997c-kube-api-access-5266s\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.464106 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.463713    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.464106 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.463774    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.464556 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.464344    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f77855-b526-401c-b8a5-3ccf9360997c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.464556 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.464499    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.464689 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.464556    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.466284 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.466258    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f77855-b526-401c-b8a5-3ccf9360997c-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.476466 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.473711    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5266s\" (UniqueName: \"kubernetes.io/projected/f8f77855-b526-401c-b8a5-3ccf9360997c-kube-api-access-5266s\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.476466 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.475720    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f77855-b526-401c-b8a5-3ccf9360997c-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.476898 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.476868    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f77855-b526-401c-b8a5-3ccf9360997c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfslf\" (UID: \"f8f77855-b526-401c-b8a5-3ccf9360997c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.499872 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.499847    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:27.590952 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.590889    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:27.865384 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.865360    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"]
Sep 09 04:05:27.867559 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:05:27.867534    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f77855_b526_401c_b8a5_3ccf9360997c.slice/crio-458592d6b45adf26792f6e7292cdca231251c2e71f38682b0ad0653311b948cd WatchSource:0}: Error finding container 458592d6b45adf26792f6e7292cdca231251c2e71f38682b0ad0653311b948cd: Status 404 returned error can't find the container with id 458592d6b45adf26792f6e7292cdca231251c2e71f38682b0ad0653311b948cd
Sep 09 04:05:27.869657 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.869627    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:05:27.869755 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.869687    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:05:27.869755 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:27.869713    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:05:28.040581 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:28.040542    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf" event={"ID":"f8f77855-b526-401c-b8a5-3ccf9360997c","Type":"ContainerStarted","Data":"a7306fc1b6b44e49495f634f65cd735cce32de590fe7b95c3712fd2d3ffd37ab"}
Sep 09 04:05:28.040766 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:28.040593    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf" event={"ID":"f8f77855-b526-401c-b8a5-3ccf9360997c","Type":"ContainerStarted","Data":"458592d6b45adf26792f6e7292cdca231251c2e71f38682b0ad0653311b948cd"}
Sep 09 04:05:28.061559 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:28.061513    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf" podStartSLOduration=1.061498894 podStartE2EDuration="1.061498894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:05:28.060356947 +0000 UTC m=+427.586727657" watchObservedRunningTime="2026-09-09 04:05:28.061498894 +0000 UTC m=+427.587869603"
Sep 09 04:05:28.591540 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:28.591511    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:28.595913 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:28.595892    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:29.044459 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:29.044419    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:29.045423 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:29.045408    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:05:40.949254 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:40.949216    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:41.063638 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.063606    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:41.119197 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.119169    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:41.275021 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.274990    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:05:41.279558 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.279539    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"
Sep 09 04:05:41.281926 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.281905    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gnr75\""
Sep 09 04:05:41.282043 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.281944    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 04:05:41.282872 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.282856    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 04:05:41.282985 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.282880    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 04:05:41.289795 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.289776    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:05:41.302801 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.302783    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:05:41.375788 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.375767    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6272166e-49cb-4c5d-8aaf-fa87f71adf02-config-file\") pod \"limitador-limitador-84994f7ffc-rbp4j\" (UID: \"6272166e-49cb-4c5d-8aaf-fa87f71adf02\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"
Sep 09 04:05:41.375909 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.375855    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp9qk\" (UniqueName: \"kubernetes.io/projected/6272166e-49cb-4c5d-8aaf-fa87f71adf02-kube-api-access-bp9qk\") pod \"limitador-limitador-84994f7ffc-rbp4j\" (UID: \"6272166e-49cb-4c5d-8aaf-fa87f71adf02\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"
Sep 09 04:05:41.477184 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.477159    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6272166e-49cb-4c5d-8aaf-fa87f71adf02-config-file\") pod \"limitador-limitador-84994f7ffc-rbp4j\" (UID: \"6272166e-49cb-4c5d-8aaf-fa87f71adf02\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"
Sep 09 04:05:41.477279 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.477208    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp9qk\" (UniqueName: \"kubernetes.io/projected/6272166e-49cb-4c5d-8aaf-fa87f71adf02-kube-api-access-bp9qk\") pod \"limitador-limitador-84994f7ffc-rbp4j\" (UID: \"6272166e-49cb-4c5d-8aaf-fa87f71adf02\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"
Sep 09 04:05:41.477822 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.477805    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6272166e-49cb-4c5d-8aaf-fa87f71adf02-config-file\") pod \"limitador-limitador-84994f7ffc-rbp4j\" (UID: \"6272166e-49cb-4c5d-8aaf-fa87f71adf02\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"
Sep 09 04:05:41.485347 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.485322    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp9qk\" (UniqueName: \"kubernetes.io/projected/6272166e-49cb-4c5d-8aaf-fa87f71adf02-kube-api-access-bp9qk\") pod \"limitador-limitador-84994f7ffc-rbp4j\" (UID: \"6272166e-49cb-4c5d-8aaf-fa87f71adf02\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"
Sep 09 04:05:41.591958 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.591888    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"
Sep 09 04:05:41.789083 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.789051    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:41.862098 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.861942    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:05:41.864234 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:05:41.864210    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6272166e_49cb_4c5d_8aaf_fa87f71adf02.slice/crio-500183f88e89ac266476e6d36e5219556d145fa01d7807c9402ea3fb3dcc904a WatchSource:0}: Error finding container 500183f88e89ac266476e6d36e5219556d145fa01d7807c9402ea3fb3dcc904a: Status 404 returned error can't find the container with id 500183f88e89ac266476e6d36e5219556d145fa01d7807c9402ea3fb3dcc904a
Sep 09 04:05:41.866341 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:41.866319    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:05:42.101313 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:42.101264    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-rbp4j" event={"ID":"6272166e-49cb-4c5d-8aaf-fa87f71adf02","Type":"ContainerStarted","Data":"500183f88e89ac266476e6d36e5219556d145fa01d7807c9402ea3fb3dcc904a"}
Sep 09 04:05:44.113513 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:44.113482    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-rbp4j" event={"ID":"6272166e-49cb-4c5d-8aaf-fa87f71adf02","Type":"ContainerStarted","Data":"44e9692e1bf04e3747e352f2dde26a2f6bd77944c104b5c24fda88e5444b90fc"}
Sep 09 04:05:44.113857 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:44.113613    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"
Sep 09 04:05:44.131253 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:44.131210    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-rbp4j" podStartSLOduration=1.7367292760000002 podStartE2EDuration="3.13119691s" totalImagesPullingTime="1.394467634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:05:44.129859084 +0000 UTC m=+443.656229795" watchObservedRunningTime="2026-09-09 04:05:44.13119691 +0000 UTC m=+443.657567619"
Sep 09 04:05:55.118756 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:05:55.118728    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"
Sep 09 04:06:00.555530 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:00.555490    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:00.655372 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:00.655348    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:01.120635 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:01.120606    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:11.375342 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:11.375307    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:11.560998 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:11.560973    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:18.847677 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:18.847646    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:18.913924 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:18.913888    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:18.968253 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:18.968221    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:19.078067 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:19.078033    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:19.200138 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:19.200072    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:21.815925 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:21.815890    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:21.916259 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:21.916217    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:34.174362 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:34.174331    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:57.003300 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:57.003268    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:57.102713 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:57.102686    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:58.607865 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:58.607835    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:58.708667 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:58.708638    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:58.900710 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:58.900637    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:59.086265 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:59.086225    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:59.179548 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:59.179462    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:06:59.307292 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:59.307263    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:06:59.318563 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:06:59.318538    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:07:08.497011 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:08.496979    2553 generic.go:356] "Generic (PLEG): container finished" podID="f8f77855-b526-401c-b8a5-3ccf9360997c" containerID="a7306fc1b6b44e49495f634f65cd735cce32de590fe7b95c3712fd2d3ffd37ab" exitCode=0
Sep 09 04:07:08.497382 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:08.497053    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf" event={"ID":"f8f77855-b526-401c-b8a5-3ccf9360997c","Type":"ContainerDied","Data":"a7306fc1b6b44e49495f634f65cd735cce32de590fe7b95c3712fd2d3ffd37ab"}
Sep 09 04:07:08.497486 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:08.497471    2553 scope.go:117] "RemoveContainer" containerID="a7306fc1b6b44e49495f634f65cd735cce32de590fe7b95c3712fd2d3ffd37ab"
Sep 09 04:07:08.498581 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:08.498556    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:07:08.498657 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:08.498640    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:07:08.498706 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:08.498690    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:07:09.506515 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:09.506474    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf" event={"ID":"f8f77855-b526-401c-b8a5-3ccf9360997c","Type":"ContainerStarted","Data":"83a112f269ddb3fc3a0a897c6b70e122e034f68b04f7d6af2b9136a6f33dd2bd"}
Sep 09 04:07:09.591211 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:09.591174    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:07:09.592054 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:09.592032    2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bfslf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 09 04:07:09.592110 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:09.592088    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf" podUID="f8f77855-b526-401c-b8a5-3ccf9360997c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 09 04:07:10.591493 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:10.591461    2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bfslf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 09 04:07:10.591870 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:10.591510    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf" podUID="f8f77855-b526-401c-b8a5-3ccf9360997c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 09 04:07:11.415508 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:11.415471    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:07:11.520325 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:11.520275    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:07:11.591570 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:11.591539    2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bfslf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 09 04:07:11.592008 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:11.591608    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf" podUID="f8f77855-b526-401c-b8a5-3ccf9360997c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 09 04:07:12.591540 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:12.591465    2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bfslf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 09 04:07:12.591540 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:12.591517    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf" podUID="f8f77855-b526-401c-b8a5-3ccf9360997c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 09 04:07:13.591606 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:13.591572    2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bfslf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Sep 09 04:07:13.591789 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:13.591621    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf" podUID="f8f77855-b526-401c-b8a5-3ccf9360997c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Sep 09 04:07:14.045335 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:14.045296    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:07:14.595656 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:14.595625    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:07:14.596709 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:14.596688    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfslf"
Sep 09 04:07:28.297358 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:28.297322    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:07:28.780000 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:28.779971    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:07:42.851126 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:42.851094    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:07:43.555675 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:07:43.555646    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:08:04.771597 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:04.771569    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"]
Sep 09 04:08:04.774166 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:04.774148    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:04.776540 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:04.776517    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 04:08:04.777744 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:04.777720    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 04:08:04.777860 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:04.777719    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 09 04:08:04.777860 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:04.777724    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6qgzz\""
Sep 09 04:08:04.782973 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:04.782947    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"]
Sep 09 04:08:04.863058 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:04.863029    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:08:04.905383 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:04.905360    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47db100f-4385-4d60-b496-1309ab3631ca-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:04.905521 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:04.905391    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47db100f-4385-4d60-b496-1309ab3631ca-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:04.905521 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:04.905422    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfpp5\" (UniqueName: \"kubernetes.io/projected/47db100f-4385-4d60-b496-1309ab3631ca-kube-api-access-xfpp5\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:04.905521 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:04.905503    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47db100f-4385-4d60-b496-1309ab3631ca-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:04.905868 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:04.905532    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47db100f-4385-4d60-b496-1309ab3631ca-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:04.905868 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:04.905567    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47db100f-4385-4d60-b496-1309ab3631ca-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:05.006724 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.006683    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47db100f-4385-4d60-b496-1309ab3631ca-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:05.006724 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.006738    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47db100f-4385-4d60-b496-1309ab3631ca-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:05.006984 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.006779    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfpp5\" (UniqueName: \"kubernetes.io/projected/47db100f-4385-4d60-b496-1309ab3631ca-kube-api-access-xfpp5\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:05.006984 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.006812    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47db100f-4385-4d60-b496-1309ab3631ca-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:05.006984 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.006836    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47db100f-4385-4d60-b496-1309ab3631ca-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:05.006984 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.006877    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47db100f-4385-4d60-b496-1309ab3631ca-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:05.007190 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.007076    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47db100f-4385-4d60-b496-1309ab3631ca-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:05.007190 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.007179    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47db100f-4385-4d60-b496-1309ab3631ca-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:05.007299 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.007265    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47db100f-4385-4d60-b496-1309ab3631ca-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:05.009067 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.009045    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47db100f-4385-4d60-b496-1309ab3631ca-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:05.009179 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.009167    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47db100f-4385-4d60-b496-1309ab3631ca-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:05.014950 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.014929    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfpp5\" (UniqueName: \"kubernetes.io/projected/47db100f-4385-4d60-b496-1309ab3631ca-kube-api-access-xfpp5\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl\" (UID: \"47db100f-4385-4d60-b496-1309ab3631ca\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:05.086184 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.086110    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:05.355905 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.355885    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"]
Sep 09 04:08:05.358402 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:08:05.358375    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47db100f_4385_4d60_b496_1309ab3631ca.slice/crio-029b3a64e66e64f7c27c5f5e7f2920e5f89e4c5128122f754b2e1c6315fc162e WatchSource:0}: Error finding container 029b3a64e66e64f7c27c5f5e7f2920e5f89e4c5128122f754b2e1c6315fc162e: Status 404 returned error can't find the container with id 029b3a64e66e64f7c27c5f5e7f2920e5f89e4c5128122f754b2e1c6315fc162e
Sep 09 04:08:05.785193 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.785157    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl" event={"ID":"47db100f-4385-4d60-b496-1309ab3631ca","Type":"ContainerStarted","Data":"029b3a64e66e64f7c27c5f5e7f2920e5f89e4c5128122f754b2e1c6315fc162e"}
Sep 09 04:08:05.861515 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:05.861479    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:08:06.792216 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:06.792186    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl" event={"ID":"47db100f-4385-4d60-b496-1309ab3631ca","Type":"ContainerStarted","Data":"8b1c06a1d781e384893be93b5e03ed0e5b6473f20a15935ee75323757c04cfe1"}
Sep 09 04:08:06.811107 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:06.811058    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl" podStartSLOduration=1.891119148 podStartE2EDuration="2.811045019s" totalImagesPullingTime="919.925871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:08:06.809467978 +0000 UTC m=+586.335838683" watchObservedRunningTime="2026-09-09 04:08:06.811045019 +0000 UTC m=+586.337415728"
Sep 09 04:08:09.396437 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:09.396408    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:08:11.313331 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:11.313292    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:08:15.086417 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:15.086384    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:15.086884 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:15.086492    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:15.099656 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:15.099635    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:15.842155 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:15.842129    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl"
Sep 09 04:08:21.190583 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:21.190553    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:08:22.557961 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:22.557924    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:08:34.976852 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:34.976814    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"]
Sep 09 04:08:34.979322 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:34.979307    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:34.981765 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:34.981741    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 04:08:34.990334 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:34.990293    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"]
Sep 09 04:08:34.997782 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:34.997757    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:08:35.151732 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.151699    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00abc0f1-4162-479b-b551-d728228711c0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.151897 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.151743    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00abc0f1-4162-479b-b551-d728228711c0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.151897 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.151762    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnnxq\" (UniqueName: \"kubernetes.io/projected/00abc0f1-4162-479b-b551-d728228711c0-kube-api-access-tnnxq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.151897 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.151826    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00abc0f1-4162-479b-b551-d728228711c0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.151897 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.151873    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00abc0f1-4162-479b-b551-d728228711c0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.152043 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.151904    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00abc0f1-4162-479b-b551-d728228711c0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.253320 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.253249    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00abc0f1-4162-479b-b551-d728228711c0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.253320 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.253299    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00abc0f1-4162-479b-b551-d728228711c0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.253546 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.253325    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnnxq\" (UniqueName: \"kubernetes.io/projected/00abc0f1-4162-479b-b551-d728228711c0-kube-api-access-tnnxq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.253546 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.253370    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00abc0f1-4162-479b-b551-d728228711c0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.253546 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.253432    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00abc0f1-4162-479b-b551-d728228711c0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.253546 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.253506    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00abc0f1-4162-479b-b551-d728228711c0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.253781 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.253753    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00abc0f1-4162-479b-b551-d728228711c0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.253847 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.253777    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00abc0f1-4162-479b-b551-d728228711c0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.253847 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.253801    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00abc0f1-4162-479b-b551-d728228711c0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.255546 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.255530    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00abc0f1-4162-479b-b551-d728228711c0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.255949 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.255929    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00abc0f1-4162-479b-b551-d728228711c0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.261516 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.261498    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnnxq\" (UniqueName: \"kubernetes.io/projected/00abc0f1-4162-479b-b551-d728228711c0-kube-api-access-tnnxq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6\" (UID: \"00abc0f1-4162-479b-b551-d728228711c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.292145 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.292124    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:35.560803 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.560781    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"]
Sep 09 04:08:35.562631 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:08:35.562605    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00abc0f1_4162_479b_b551_d728228711c0.slice/crio-5086ce3d8a9e7898a0c0c961341e0703255e7d8e8bcf5fe3cf90d862db3e36d1 WatchSource:0}: Error finding container 5086ce3d8a9e7898a0c0c961341e0703255e7d8e8bcf5fe3cf90d862db3e36d1: Status 404 returned error can't find the container with id 5086ce3d8a9e7898a0c0c961341e0703255e7d8e8bcf5fe3cf90d862db3e36d1
Sep 09 04:08:35.563992 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.563972    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 04:08:35.938360 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.938280    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6" event={"ID":"00abc0f1-4162-479b-b551-d728228711c0","Type":"ContainerStarted","Data":"5d502eb0fea33dfe3f2ed41fb85e6207360d89355e98a795e7648e21227b6874"}
Sep 09 04:08:35.938360 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.938347    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6" event={"ID":"00abc0f1-4162-479b-b551-d728228711c0","Type":"ContainerStarted","Data":"5086ce3d8a9e7898a0c0c961341e0703255e7d8e8bcf5fe3cf90d862db3e36d1"}
Sep 09 04:08:35.955752 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:35.955703    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6" podStartSLOduration=1.78879448 podStartE2EDuration="1.955687646s" totalImagesPullingTime="166.893166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:08:35.954380615 +0000 UTC m=+615.480751336" watchObservedRunningTime="2026-09-09 04:08:35.955687646 +0000 UTC m=+615.482058356"
Sep 09 04:08:36.652800 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:36.652770    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:08:43.664457 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:43.664351    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:08:43.740621 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:43.740592    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:08:45.293249 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:45.293215    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:45.293740 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:45.293304    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:45.305531 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:45.305510    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:46.001640 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:46.001614    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6"
Sep 09 04:08:52.659424 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.659389    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"]
Sep 09 04:08:52.662677 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.662660    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.665156 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.665130    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 09 04:08:52.672138 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.672116    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"]
Sep 09 04:08:52.693643 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.693622    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:08:52.693965 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.693950    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kt7g\" (UniqueName: \"kubernetes.io/projected/605df143-78f1-4424-ba50-b28c94bb0722-kube-api-access-6kt7g\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.694014 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.693985    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605df143-78f1-4424-ba50-b28c94bb0722-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.694063 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.694029    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605df143-78f1-4424-ba50-b28c94bb0722-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.694063 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.694049    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605df143-78f1-4424-ba50-b28c94bb0722-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.694147 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.694132    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605df143-78f1-4424-ba50-b28c94bb0722-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.694185 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.694151    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605df143-78f1-4424-ba50-b28c94bb0722-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.795189 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.795168    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605df143-78f1-4424-ba50-b28c94bb0722-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.795327 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.795196    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605df143-78f1-4424-ba50-b28c94bb0722-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.795327 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.795245    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kt7g\" (UniqueName: \"kubernetes.io/projected/605df143-78f1-4424-ba50-b28c94bb0722-kube-api-access-6kt7g\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.795327 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.795271    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605df143-78f1-4424-ba50-b28c94bb0722-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.795327 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.795310    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605df143-78f1-4424-ba50-b28c94bb0722-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.795537 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.795327    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605df143-78f1-4424-ba50-b28c94bb0722-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.795711 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.795681    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605df143-78f1-4424-ba50-b28c94bb0722-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.795772 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.795739    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605df143-78f1-4424-ba50-b28c94bb0722-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.795772 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.795763    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605df143-78f1-4424-ba50-b28c94bb0722-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.797650 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.797632    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605df143-78f1-4424-ba50-b28c94bb0722-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.797809 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.797794    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605df143-78f1-4424-ba50-b28c94bb0722-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.805673 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.805651    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kt7g\" (UniqueName: \"kubernetes.io/projected/605df143-78f1-4424-ba50-b28c94bb0722-kube-api-access-6kt7g\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gfd5t\" (UID: \"605df143-78f1-4424-ba50-b28c94bb0722\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:52.976151 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:52.976123    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:08:53.246954 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:53.246930    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"]
Sep 09 04:08:53.248943 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:08:53.248913    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605df143_78f1_4424_ba50_b28c94bb0722.slice/crio-c6991a0e8f871b0f5713c313851ca0fef9f4be9902e576bb5d1156d1893eb4c1 WatchSource:0}: Error finding container c6991a0e8f871b0f5713c313851ca0fef9f4be9902e576bb5d1156d1893eb4c1: Status 404 returned error can't find the container with id c6991a0e8f871b0f5713c313851ca0fef9f4be9902e576bb5d1156d1893eb4c1
Sep 09 04:08:54.029878 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:54.029832    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t" event={"ID":"605df143-78f1-4424-ba50-b28c94bb0722","Type":"ContainerStarted","Data":"37e9c33d4c6422f933a36e05a164fddea94e872c3ff65ef797579cd5144a534d"}
Sep 09 04:08:54.030310 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:54.029904    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t" event={"ID":"605df143-78f1-4424-ba50-b28c94bb0722","Type":"ContainerStarted","Data":"c6991a0e8f871b0f5713c313851ca0fef9f4be9902e576bb5d1156d1893eb4c1"}
Sep 09 04:08:54.048836 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:54.047163    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t" podStartSLOduration=1.854093368 podStartE2EDuration="2.047144392s" totalImagesPullingTime="193.051024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:08:54.045759081 +0000 UTC m=+633.572129793" watchObservedRunningTime="2026-09-09 04:08:54.047144392 +0000 UTC m=+633.573515103"
Sep 09 04:08:54.755381 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:08:54.755349    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:09:02.976478 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:02.976426    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:09:02.976901 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:02.976500    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:09:02.989028 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:02.989000    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:09:03.083268 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:03.083241    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gfd5t"
Sep 09 04:09:05.610993 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:05.610963    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:09:07.442281 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.442246    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"]
Sep 09 04:09:07.452981 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.452962    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.455652 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.455629    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 04:09:07.455868 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.455850    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"]
Sep 09 04:09:07.475802 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.475777    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:09:07.513719 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.513696    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd55cd03-51e6-4481-9ba9-bd656969ca62-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.513826 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.513731    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd55cd03-51e6-4481-9ba9-bd656969ca62-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.513902 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.513820    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd55cd03-51e6-4481-9ba9-bd656969ca62-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.513965 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.513917    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd55cd03-51e6-4481-9ba9-bd656969ca62-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.513965 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.513948    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqxv6\" (UniqueName: \"kubernetes.io/projected/cd55cd03-51e6-4481-9ba9-bd656969ca62-kube-api-access-zqxv6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.514051 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.513979    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd55cd03-51e6-4481-9ba9-bd656969ca62-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.615122 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.615097    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd55cd03-51e6-4481-9ba9-bd656969ca62-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.615253 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.615129    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqxv6\" (UniqueName: \"kubernetes.io/projected/cd55cd03-51e6-4481-9ba9-bd656969ca62-kube-api-access-zqxv6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.615253 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.615155    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd55cd03-51e6-4481-9ba9-bd656969ca62-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.615253 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.615189    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd55cd03-51e6-4481-9ba9-bd656969ca62-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.615253 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.615235    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd55cd03-51e6-4481-9ba9-bd656969ca62-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.615461 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.615301    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd55cd03-51e6-4481-9ba9-bd656969ca62-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.615560 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.615543    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd55cd03-51e6-4481-9ba9-bd656969ca62-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.615625 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.615609    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd55cd03-51e6-4481-9ba9-bd656969ca62-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.615694 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.615676    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd55cd03-51e6-4481-9ba9-bd656969ca62-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.617572 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.617553    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd55cd03-51e6-4481-9ba9-bd656969ca62-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.617894 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.617876    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd55cd03-51e6-4481-9ba9-bd656969ca62-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.622771 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.622751    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqxv6\" (UniqueName: \"kubernetes.io/projected/cd55cd03-51e6-4481-9ba9-bd656969ca62-kube-api-access-zqxv6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8\" (UID: \"cd55cd03-51e6-4481-9ba9-bd656969ca62\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:07.764096 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:07.764071    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:08.035374 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:08.035351    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"]
Sep 09 04:09:08.037591 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:09:08.037569    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd55cd03_51e6_4481_9ba9_bd656969ca62.slice/crio-d9c381732f703285f51775d3ff72f1a936753ce06d07ed31398d36a8b2580528 WatchSource:0}: Error finding container d9c381732f703285f51775d3ff72f1a936753ce06d07ed31398d36a8b2580528: Status 404 returned error can't find the container with id d9c381732f703285f51775d3ff72f1a936753ce06d07ed31398d36a8b2580528
Sep 09 04:09:08.095997 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:08.095973    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8" event={"ID":"cd55cd03-51e6-4481-9ba9-bd656969ca62","Type":"ContainerStarted","Data":"d9c381732f703285f51775d3ff72f1a936753ce06d07ed31398d36a8b2580528"}
Sep 09 04:09:09.101808 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:09.101773    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8" event={"ID":"cd55cd03-51e6-4481-9ba9-bd656969ca62","Type":"ContainerStarted","Data":"c6501b63d2f062faa0705c6510f2a75945b85bf6cd7934e6a2404eadc33f5e65"}
Sep 09 04:09:09.118831 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:09.118780    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8" podStartSLOduration=1.9327193729999999 podStartE2EDuration="2.118761316s" totalImagesPullingTime="186.041943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:09:09.118219915 +0000 UTC m=+648.644590629" watchObservedRunningTime="2026-09-09 04:09:09.118761316 +0000 UTC m=+648.645132026"
Sep 09 04:09:09.758803 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:09.758767    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:09:17.764913 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:17.764881    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:17.765325 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:17.764929    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:17.777064 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:17.777043    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:18.155247 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:18.155178    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8"
Sep 09 04:09:28.038657 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:28.038623    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:09:28.139283 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:28.139256    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:09:55.832994 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:09:55.832950    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:18.033081 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:18.033002    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:18.132401 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:18.132369    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:19.157434 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:19.157408    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:19.257406 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:19.257378    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:35.491134 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:35.491102    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:35.595463 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:35.595421    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:35.693846 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:35.693816    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:35.820812 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:35.820743    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:35.900784 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:35.900742    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:35.966568 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:35.966542    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:36.117472 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:36.117368    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:36.929816 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:36.929785    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:37.030742 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:37.030710    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:37.396191 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:37.396166    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:39.963056 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:39.963026    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:10:41.528764 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:41.528726    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:42.322629 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:42.322597    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:42.424508 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:42.424430    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:42.809815 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:42.809790    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:42.912145 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:42.912106    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:43.244586 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:43.244548    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:43.350590 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:43.350565    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:43.431575 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:43.431545    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:43.800186 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:43.800157    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:43.871568 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:43.871538    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:44.275268 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.275238    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:44.391156 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.391122    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:44.487625 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.487586    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:44.655427 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.655347    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"]
Sep 09 04:10:44.658901 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.658831    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.661111 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.661090    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-4ad5fa-openshift-default-dockercfg-l9vvl\""
Sep 09 04:10:44.669203 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.669184    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"]
Sep 09 04:10:44.680507 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.680484    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:44.783393 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.783369    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:44.821648 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.821622    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istiod-ca-cert\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.821745 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.821655    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-credential-socket\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.821745 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.821678    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-workload-certs\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.821745 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.821736    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9b8s\" (UniqueName: \"kubernetes.io/projected/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-kube-api-access-s9b8s\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.821863 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.821764    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-workload-socket\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.821863 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.821806    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-podinfo\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.821863 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.821842    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-data\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.821979 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.821871    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-token\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.821979 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.821900    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-envoy\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.883370 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.883349    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:44.923123 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.923051    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-podinfo\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.923123 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.923093    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-data\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.923123 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.923124    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-token\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.923346 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.923155    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-envoy\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.923346 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.923189    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istiod-ca-cert\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.923346 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.923218    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-credential-socket\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.923346 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.923251    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-workload-certs\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.923346 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.923287    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9b8s\" (UniqueName: \"kubernetes.io/projected/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-kube-api-access-s9b8s\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.923637 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.923485    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-workload-socket\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.923637 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.923522    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-data\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.923747 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.923682    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-credential-socket\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.923815 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.923799    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-workload-socket\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.924025 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.924010    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-workload-certs\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.924173 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.924150    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istiod-ca-cert\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.925576 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.925557    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-envoy\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.925783 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.925766    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-podinfo\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.930678 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.930652    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-token\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.930895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.930876    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9b8s\" (UniqueName: \"kubernetes.io/projected/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-kube-api-access-s9b8s\") pod \"e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") " pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:44.970035 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:44.970014    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:45.243453 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:45.243411    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"]
Sep 09 04:10:45.245091 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:10:45.245065    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02557a9e_4916_4bdc_a1f2_b76f241ca2b0.slice/crio-48066ab651e485ed543573c3f57ffd2d65b81b9a754a31e4ae420e6ad426e7c4 WatchSource:0}: Error finding container 48066ab651e485ed543573c3f57ffd2d65b81b9a754a31e4ae420e6ad426e7c4: Status 404 returned error can't find the container with id 48066ab651e485ed543573c3f57ffd2d65b81b9a754a31e4ae420e6ad426e7c4
Sep 09 04:10:45.247412 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:45.247375    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:10:45.247525 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:45.247473    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:10:45.247590 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:45.247527    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:10:45.551299 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:45.551214    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq" event={"ID":"02557a9e-4916-4bdc-a1f2-b76f241ca2b0","Type":"ContainerStarted","Data":"95d5a3d6bf7145498f45729ca9323c4bb5842c1e28dc64a5cdcb2bc3d0522a66"}
Sep 09 04:10:45.551299 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:45.551264    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq" event={"ID":"02557a9e-4916-4bdc-a1f2-b76f241ca2b0","Type":"ContainerStarted","Data":"48066ab651e485ed543573c3f57ffd2d65b81b9a754a31e4ae420e6ad426e7c4"}
Sep 09 04:10:45.572822 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:45.572770    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq" podStartSLOduration=1.572755463 podStartE2EDuration="1.572755463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:10:45.571282018 +0000 UTC m=+745.097652751" watchObservedRunningTime="2026-09-09 04:10:45.572755463 +0000 UTC m=+745.099126173"
Sep 09 04:10:45.609036 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:45.609005    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:45.703398 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:45.703373    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:45.757860 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:45.757815    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:45.860695 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:45.860616    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:45.971077 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:45.971051    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:46.170380 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.170314    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"]
Sep 09 04:10:46.173731 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.173715    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.177297 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.177256    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-45a82a13-openshift-default-dockercfg-g4q9n\""
Sep 09 04:10:46.188123 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.188105    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"]
Sep 09 04:10:46.203146 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.203121    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:46.339213 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.339181    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-workload-certs\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.339382 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.339234    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-data\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.339382 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.339270    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-podinfo\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.339382 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.339316    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istiod-ca-cert\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.339382 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.339338    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-workload-socket\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.339382 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.339377    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-envoy\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.339617 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.339395    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-token\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.339617 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.339498    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-credential-socket\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.339617 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.339526    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prds7\" (UniqueName: \"kubernetes.io/projected/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-kube-api-access-prds7\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.410565 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.410533    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:46.440463 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.440378    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-envoy\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.440463 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.440406    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-token\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.440463 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.440428    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-credential-socket\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.440673 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.440485    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prds7\" (UniqueName: \"kubernetes.io/projected/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-kube-api-access-prds7\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.440673 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.440519    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-workload-certs\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.440673 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.440577    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-data\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.440673 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.440630    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-podinfo\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.440673 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.440670    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istiod-ca-cert\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.440926 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.440699    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-workload-socket\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.440926 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.440911    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-credential-socket\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.441018 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.440987    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-data\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.441061 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.441014    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-workload-certs\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.441061 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.441040    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-workload-socket\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.441391 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.441362    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istiod-ca-cert\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.442812 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.442794    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-envoy\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.443177 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.443162    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-podinfo\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.448616 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.448590    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-token\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.448689 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.448663    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prds7\" (UniqueName: \"kubernetes.io/projected/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-kube-api-access-prds7\") pod \"e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") " pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.484813 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.484794    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:46.761547 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.761523    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"]
Sep 09 04:10:46.763188 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:10:46.763164    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fce7a56_ef91_4ae4_83f6_f3ec809d55d1.slice/crio-01e9581615f235c6cf30d23df911fd2aafa2d6ab510798b8775460cfca2de7c2 WatchSource:0}: Error finding container 01e9581615f235c6cf30d23df911fd2aafa2d6ab510798b8775460cfca2de7c2: Status 404 returned error can't find the container with id 01e9581615f235c6cf30d23df911fd2aafa2d6ab510798b8775460cfca2de7c2
Sep 09 04:10:46.765450 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.765408    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:10:46.765527 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.765486    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:10:46.765527 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.765516    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:10:46.975342 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:46.975307    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:47.563507 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:47.563468    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk" event={"ID":"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1","Type":"ContainerStarted","Data":"8648d7e8287c2bb1246c7331cd72b00d4b59cfb545f7d1b2600461a8754b50d3"}
Sep 09 04:10:47.563713 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:47.563528    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk" event={"ID":"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1","Type":"ContainerStarted","Data":"01e9581615f235c6cf30d23df911fd2aafa2d6ab510798b8775460cfca2de7c2"}
Sep 09 04:10:47.563713 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:47.563670    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:47.565000 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:47.564974    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:10:47.584156 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:47.584108    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk" podStartSLOduration=1.584091352 podStartE2EDuration="1.584091352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:10:47.581602127 +0000 UTC m=+747.107972837" watchObservedRunningTime="2026-09-09 04:10:47.584091352 +0000 UTC m=+747.110462064"
Sep 09 04:10:47.623775 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:47.623736    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:47.719024 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:47.718996    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:48.485878 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:48.485847    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:48.490884 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:48.490851    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:48.568710 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:48.568649    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:48.569654 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:48.569619    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:10:48.799886 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:48.799801    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:48.903856 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:48.903826    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:49.006591 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:49.006555    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:49.104138 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:49.104062    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:49.204432 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:49.204408    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:49.479631 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:49.479588    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:49.579081 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:49.579057    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:50.497650 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:50.497613    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:50.598318 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:50.598262    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:51.040435 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.040403    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:51.129425 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.129395    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:51.242065 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.242034    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"]
Sep 09 04:10:51.245653 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.245638    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"
Sep 09 04:10:51.247874 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.247856    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-45a82a13\""
Sep 09 04:10:51.247962 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.247898    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-45a82a13-dockercfg-cswt5\""
Sep 09 04:10:51.254789 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.254765    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"]
Sep 09 04:10:51.287245 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.287222    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxkw7\" (UniqueName: \"kubernetes.io/projected/141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8-kube-api-access-wxkw7\") pod \"payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf\" (UID: \"141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"
Sep 09 04:10:51.287346 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.287278    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf\" (UID: \"141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"
Sep 09 04:10:51.388687 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.388608    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxkw7\" (UniqueName: \"kubernetes.io/projected/141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8-kube-api-access-wxkw7\") pod \"payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf\" (UID: \"141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"
Sep 09 04:10:51.388687 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.388659    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf\" (UID: \"141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"
Sep 09 04:10:51.389327 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.389309    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf\" (UID: \"141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"
Sep 09 04:10:51.396997 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.396973    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxkw7\" (UniqueName: \"kubernetes.io/projected/141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8-kube-api-access-wxkw7\") pod \"payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf\" (UID: \"141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"
Sep 09 04:10:51.557163 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.557139    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"
Sep 09 04:10:51.824770 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:51.824749    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"]
Sep 09 04:10:51.827365 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:10:51.827335    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141e64cd_bf2a_4ff1_bfca_fbc5b5792bf8.slice/crio-89e01df6407206e3f84dd9f0da4f1d5f8bbc7b43f18aceeff7d921e5ae0252db WatchSource:0}: Error finding container 89e01df6407206e3f84dd9f0da4f1d5f8bbc7b43f18aceeff7d921e5ae0252db: Status 404 returned error can't find the container with id 89e01df6407206e3f84dd9f0da4f1d5f8bbc7b43f18aceeff7d921e5ae0252db
Sep 09 04:10:52.593630 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:52.593580    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf" event={"ID":"141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8","Type":"ContainerStarted","Data":"89e01df6407206e3f84dd9f0da4f1d5f8bbc7b43f18aceeff7d921e5ae0252db"}
Sep 09 04:10:53.601876 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:53.601852    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf" event={"ID":"141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8","Type":"ContainerStarted","Data":"c5455c074e8646fc3ec407f079d44b7f59727b3213a39f1d3154425bca2222bb"}
Sep 09 04:10:53.602260 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:53.601970    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"
Sep 09 04:10:53.618081 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:53.618038    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf" podStartSLOduration=0.919384586 podStartE2EDuration="2.618023545s" totalImagesPullingTime="1.698638959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:10:53.616871136 +0000 UTC m=+753.143241850" watchObservedRunningTime="2026-09-09 04:10:53.618023545 +0000 UTC m=+753.144394255"
Sep 09 04:10:56.215460 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:56.215385    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:56.322504 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:56.322468    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:56.424312 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:56.424283    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:56.526772 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:56.526745    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:10:57.368148 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:57.368116    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:10:59.231987 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:10:59.231946    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:00.999175 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:00.999137    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:03.038936 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:03.038884    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:03.148090 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:03.147858    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:03.784391 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:03.784354    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:04.609632 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:04.609604    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"
Sep 09 04:11:05.213467 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:05.213029    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:05.995201 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:05.995155    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:06.483866 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:06.483814    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:06.666866 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:06.666827    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:06.914620 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:06.914532    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:07.010405 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:07.010358    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:07.105643 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:07.105609    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:07.210812 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:07.210732    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:07.697455 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:07.697410    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:07.948514 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:07.948414    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:09.551526 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.551492    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:09.608921 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.608894    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:09.686361 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.686338    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:09.693586 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.693564    2553 generic.go:356] "Generic (PLEG): container finished" podID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" containerID="95d5a3d6bf7145498f45729ca9323c4bb5842c1e28dc64a5cdcb2bc3d0522a66" exitCode=0
Sep 09 04:11:09.693704 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.693642    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq" event={"ID":"02557a9e-4916-4bdc-a1f2-b76f241ca2b0","Type":"ContainerDied","Data":"95d5a3d6bf7145498f45729ca9323c4bb5842c1e28dc64a5cdcb2bc3d0522a66"}
Sep 09 04:11:09.694060 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.694039    2553 scope.go:117] "RemoveContainer" containerID="95d5a3d6bf7145498f45729ca9323c4bb5842c1e28dc64a5cdcb2bc3d0522a66"
Sep 09 04:11:09.695321 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.695301    2553 generic.go:356] "Generic (PLEG): container finished" podID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerID="8648d7e8287c2bb1246c7331cd72b00d4b59cfb545f7d1b2600461a8754b50d3" exitCode=0
Sep 09 04:11:09.695391 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.695301    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:11:09.695391 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.695363    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk" event={"ID":"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1","Type":"ContainerDied","Data":"8648d7e8287c2bb1246c7331cd72b00d4b59cfb545f7d1b2600461a8754b50d3"}
Sep 09 04:11:09.695535 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.695404    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:11:09.695535 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.695464    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:11:09.695732 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.695717    2553 scope.go:117] "RemoveContainer" containerID="8648d7e8287c2bb1246c7331cd72b00d4b59cfb545f7d1b2600461a8754b50d3"
Sep 09 04:11:09.696774 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.696751    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:11:09.696837 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.696805    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:11:09.696904 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.696890    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:11:09.785505 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.785482    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:09.913076 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:09.912981    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:10.010434 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:10.010402    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:10.707616 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:10.707574    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq" event={"ID":"02557a9e-4916-4bdc-a1f2-b76f241ca2b0","Type":"ContainerStarted","Data":"428f2056d4c1750d12152bdf15d04c1979d38c718002c93947e62207da303246"}
Sep 09 04:11:10.711271 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:10.711234    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk" event={"ID":"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1","Type":"ContainerStarted","Data":"d3606989e07a5e81cc3ef56cd78ddb893eab9c0248a2dbd9809de5383aafeb4b"}
Sep 09 04:11:10.970535 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:10.970502    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:11:10.970839 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:10.970816    2553 patch_prober.go:28] interesting pod/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 09 04:11:10.970909 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:10.970888    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq" podUID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 09 04:11:11.001007 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:11.000980    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:11.098772 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:11.098736    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:11.485220 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:11.485190    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:11:11.485411 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:11.485294    2553 patch_prober.go:28] interesting pod/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 04:11:11.485411 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:11.485348    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk" podUID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 04:11:11.970950 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:11.970912    2553 patch_prober.go:28] interesting pod/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 09 04:11:11.971406 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:11.970986    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq" podUID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 09 04:11:12.135370 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.135312    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:12.193853 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.193811    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:12.419192 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.419098    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:12.485227 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.485196    2553 patch_prober.go:28] interesting pod/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 04:11:12.485377 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.485243    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk" podUID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 04:11:12.559081 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.559043    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:12.665575 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.665535    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:12.678176 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.678107    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:11:12.763801 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.763771    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:12.824469 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.823761    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"]
Sep 09 04:11:12.829663 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.828139    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"
Sep 09 04:11:12.833100 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.833072    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"]
Sep 09 04:11:12.833882 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.833862    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-629522-dockercfg-tbw4s\""
Sep 09 04:11:12.834345 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.834153    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-629522\""
Sep 09 04:11:12.918067 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.918025    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqj26\" (UniqueName: \"kubernetes.io/projected/4fecc125-b7a9-407f-a168-14f39568fc47-kube-api-access-tqj26\") pod \"payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4\" (UID: \"4fecc125-b7a9-407f-a168-14f39568fc47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"
Sep 09 04:11:12.918333 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.918314    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fecc125-b7a9-407f-a168-14f39568fc47-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4\" (UID: \"4fecc125-b7a9-407f-a168-14f39568fc47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"
Sep 09 04:11:12.971349 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.971321    2553 patch_prober.go:28] interesting pod/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 09 04:11:12.971873 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:12.971842    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq" podUID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 09 04:11:13.019777 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.019741    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqj26\" (UniqueName: \"kubernetes.io/projected/4fecc125-b7a9-407f-a168-14f39568fc47-kube-api-access-tqj26\") pod \"payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4\" (UID: \"4fecc125-b7a9-407f-a168-14f39568fc47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"
Sep 09 04:11:13.019942 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.019800    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fecc125-b7a9-407f-a168-14f39568fc47-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4\" (UID: \"4fecc125-b7a9-407f-a168-14f39568fc47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"
Sep 09 04:11:13.020744 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.020719    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fecc125-b7a9-407f-a168-14f39568fc47-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4\" (UID: \"4fecc125-b7a9-407f-a168-14f39568fc47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"
Sep 09 04:11:13.033089 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.033059    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqj26\" (UniqueName: \"kubernetes.io/projected/4fecc125-b7a9-407f-a168-14f39568fc47-kube-api-access-tqj26\") pod \"payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4\" (UID: \"4fecc125-b7a9-407f-a168-14f39568fc47\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"
Sep 09 04:11:13.096688 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.096659    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:13.162426 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.162397    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"
Sep 09 04:11:13.482349 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.482327    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"]
Sep 09 04:11:13.484618 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:11:13.484592    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fecc125_b7a9_407f_a168_14f39568fc47.slice/crio-21a01ab4cf6ff115e256b3e52ea8b998cbd2bec6164f4d8a4378dd3698ef9c18 WatchSource:0}: Error finding container 21a01ab4cf6ff115e256b3e52ea8b998cbd2bec6164f4d8a4378dd3698ef9c18: Status 404 returned error can't find the container with id 21a01ab4cf6ff115e256b3e52ea8b998cbd2bec6164f4d8a4378dd3698ef9c18
Sep 09 04:11:13.485610 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.485591    2553 patch_prober.go:28] interesting pod/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 04:11:13.485703 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.485632    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk" podUID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 04:11:13.763955 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.763915    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4" event={"ID":"4fecc125-b7a9-407f-a168-14f39568fc47","Type":"ContainerStarted","Data":"15cbf34c9da73e0dfad034f2d8812aa36c410845625a9220f270ecb52b356aae"}
Sep 09 04:11:13.764131 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.764015    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"
Sep 09 04:11:13.764131 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.764041    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4" event={"ID":"4fecc125-b7a9-407f-a168-14f39568fc47","Type":"ContainerStarted","Data":"21a01ab4cf6ff115e256b3e52ea8b998cbd2bec6164f4d8a4378dd3698ef9c18"}
Sep 09 04:11:13.780465 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.780369    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4" podStartSLOduration=1.78035577 podStartE2EDuration="1.78035577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:11:13.778884322 +0000 UTC m=+773.305255032" watchObservedRunningTime="2026-09-09 04:11:13.78035577 +0000 UTC m=+773.306726479"
Sep 09 04:11:13.970516 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.970483    2553 patch_prober.go:28] interesting pod/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 09 04:11:13.970691 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:13.970544    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq" podUID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 09 04:11:14.005633 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:14.005602    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:14.485524 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:14.485490    2553 patch_prober.go:28] interesting pod/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 04:11:14.485701 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:14.485537    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk" podUID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 04:11:14.885575 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:14.885419    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:14.971291 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:14.971255    2553 patch_prober.go:28] interesting pod/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 09 04:11:14.971472 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:14.971311    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq" podUID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 09 04:11:15.485616 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:15.485583    2553 patch_prober.go:28] interesting pod/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 04:11:15.486014 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:15.485647    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk" podUID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 04:11:15.977620 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:15.977591    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:11:15.978046 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:15.978021    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:11:15.979146 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:15.979119    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:11:16.489417 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:16.489391    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:11:16.489789 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:16.489669    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:11:16.490384 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:16.490367    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:11:16.689486 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:16.689459    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:16.787376 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:16.787297    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:19.679436 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:19.679398    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz"]
Sep 09 04:11:19.685534 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:19.685496    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz"
Sep 09 04:11:19.688326 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:19.688303    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:11:19.688490 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:19.688436    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:11:19.688749 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:19.688727    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz"]
Sep 09 04:11:19.801592 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:19.801568    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e04c640-f330-4114-a731-e213b88b8e08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45a82a13-952dz\" (UID: \"5e04c640-f330-4114-a731-e213b88b8e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz"
Sep 09 04:11:19.902155 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:19.902127    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e04c640-f330-4114-a731-e213b88b8e08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45a82a13-952dz\" (UID: \"5e04c640-f330-4114-a731-e213b88b8e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz"
Sep 09 04:11:19.902775 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:19.902756    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e04c640-f330-4114-a731-e213b88b8e08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45a82a13-952dz\" (UID: \"5e04c640-f330-4114-a731-e213b88b8e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz"
Sep 09 04:11:19.997885 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:19.997859    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz"
Sep 09 04:11:20.268842 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:20.268817    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz"]
Sep 09 04:11:20.271243 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:11:20.271217    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e04c640_f330_4114_a731_e213b88b8e08.slice/crio-5d5ef2dbd9d50845c75fc846a9bed541422746098c4fb1914f4e42bd3aa7d3ae WatchSource:0}: Error finding container 5d5ef2dbd9d50845c75fc846a9bed541422746098c4fb1914f4e42bd3aa7d3ae: Status 404 returned error can't find the container with id 5d5ef2dbd9d50845c75fc846a9bed541422746098c4fb1914f4e42bd3aa7d3ae
Sep 09 04:11:20.802496 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:20.802463    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz" event={"ID":"5e04c640-f330-4114-a731-e213b88b8e08","Type":"ContainerStarted","Data":"5d5ef2dbd9d50845c75fc846a9bed541422746098c4fb1914f4e42bd3aa7d3ae"}
Sep 09 04:11:22.817538 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:22.817503    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz" event={"ID":"5e04c640-f330-4114-a731-e213b88b8e08","Type":"ContainerStarted","Data":"6345fa4ecd2055ce75045dcecf3de1639fd4b208872a64dd8aa75ca301fb5af6"}
Sep 09 04:11:22.834896 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:22.834851    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz" podStartSLOduration=1.693933229 podStartE2EDuration="3.834836176s" totalImagesPullingTime="2.140902947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:11:22.833639226 +0000 UTC m=+782.360009935" watchObservedRunningTime="2026-09-09 04:11:22.834836176 +0000 UTC m=+782.361206885"
Sep 09 04:11:23.824652 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:23.824614    2553 generic.go:356] "Generic (PLEG): container finished" podID="5e04c640-f330-4114-a731-e213b88b8e08" containerID="6345fa4ecd2055ce75045dcecf3de1639fd4b208872a64dd8aa75ca301fb5af6" exitCode=0
Sep 09 04:11:23.824999 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:23.824693    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz" event={"ID":"5e04c640-f330-4114-a731-e213b88b8e08","Type":"ContainerDied","Data":"6345fa4ecd2055ce75045dcecf3de1639fd4b208872a64dd8aa75ca301fb5af6"}
Sep 09 04:11:24.770587 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:24.770561    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"
Sep 09 04:11:24.953974 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:24.953953    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz"
Sep 09 04:11:25.149962 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:25.149869    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e04c640-f330-4114-a731-e213b88b8e08-maas-api-service-ca\") pod \"5e04c640-f330-4114-a731-e213b88b8e08\" (UID: \"5e04c640-f330-4114-a731-e213b88b8e08\") "
Sep 09 04:11:25.150216 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:25.150195    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e04c640-f330-4114-a731-e213b88b8e08-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e04c640-f330-4114-a731-e213b88b8e08" (UID: "5e04c640-f330-4114-a731-e213b88b8e08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:11:25.250735 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:25.250709    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e04c640-f330-4114-a731-e213b88b8e08-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:11:25.836900 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:25.836866    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz" event={"ID":"5e04c640-f330-4114-a731-e213b88b8e08","Type":"ContainerDied","Data":"5d5ef2dbd9d50845c75fc846a9bed541422746098c4fb1914f4e42bd3aa7d3ae"}
Sep 09 04:11:25.837091 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:25.836904    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz"
Sep 09 04:11:25.837091 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:25.836936    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d5ef2dbd9d50845c75fc846a9bed541422746098c4fb1914f4e42bd3aa7d3ae"
Sep 09 04:11:30.870190 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:30.870161    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:11:30.941670 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:30.941637    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:31.003128 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.003100    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"]
Sep 09 04:11:31.003332 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.003301    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf" podUID="141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8" containerName="payload-pre-processing" containerID="cri-o://c5455c074e8646fc3ec407f079d44b7f59727b3213a39f1d3154425bca2222bb" gracePeriod=30
Sep 09 04:11:31.024845 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.024821    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:31.076617 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.076589    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:31.179121 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.179090    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:31.179958 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.179938    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz"]
Sep 09 04:11:31.183253 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.183232    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45a82a13-952dz"]
Sep 09 04:11:31.251837 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.251815    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"
Sep 09 04:11:31.273578 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.273546    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:31.299669 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.299636    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxkw7\" (UniqueName: \"kubernetes.io/projected/141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8-kube-api-access-wxkw7\") pod \"141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8\" (UID: \"141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8\") "
Sep 09 04:11:31.299826 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.299720    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8-plugins-config-volume\") pod \"141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8\" (UID: \"141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8\") "
Sep 09 04:11:31.300384 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.300350    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8" (UID: "141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:11:31.304946 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.304903    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8-kube-api-access-wxkw7" (OuterVolumeSpecName: "kube-api-access-wxkw7") pod "141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8" (UID: "141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8"). InnerVolumeSpecName "kube-api-access-wxkw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:11:31.319990 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.319966    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"]
Sep 09 04:11:31.320657 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.320205    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk" podUID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerName="istio-proxy" containerID="cri-o://d3606989e07a5e81cc3ef56cd78ddb893eab9c0248a2dbd9809de5383aafeb4b" gracePeriod=30
Sep 09 04:11:31.343261 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.343232    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:31.400723 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.400697    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxkw7\" (UniqueName: \"kubernetes.io/projected/141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8-kube-api-access-wxkw7\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:11:31.400723 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.400717    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:11:31.491772 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.491740    2553 patch_prober.go:28] interesting pod/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 04:11:31.491936 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.491790    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk" podUID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 04:11:31.821613 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.821534    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:31.869340 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.869302    2553 generic.go:356] "Generic (PLEG): container finished" podID="141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8" containerID="c5455c074e8646fc3ec407f079d44b7f59727b3213a39f1d3154425bca2222bb" exitCode=0
Sep 09 04:11:31.869485 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.869355    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"
Sep 09 04:11:31.869485 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.869383    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf" event={"ID":"141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8","Type":"ContainerDied","Data":"c5455c074e8646fc3ec407f079d44b7f59727b3213a39f1d3154425bca2222bb"}
Sep 09 04:11:31.869485 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.869433    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf" event={"ID":"141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8","Type":"ContainerDied","Data":"89e01df6407206e3f84dd9f0da4f1d5f8bbc7b43f18aceeff7d921e5ae0252db"}
Sep 09 04:11:31.869485 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.869468    2553 scope.go:117] "RemoveContainer" containerID="c5455c074e8646fc3ec407f079d44b7f59727b3213a39f1d3154425bca2222bb"
Sep 09 04:11:31.878584 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.878566    2553 scope.go:117] "RemoveContainer" containerID="c5455c074e8646fc3ec407f079d44b7f59727b3213a39f1d3154425bca2222bb"
Sep 09 04:11:31.878857 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:11:31.878811    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5455c074e8646fc3ec407f079d44b7f59727b3213a39f1d3154425bca2222bb\": container with ID starting with c5455c074e8646fc3ec407f079d44b7f59727b3213a39f1d3154425bca2222bb not found: ID does not exist" containerID="c5455c074e8646fc3ec407f079d44b7f59727b3213a39f1d3154425bca2222bb"
Sep 09 04:11:31.878857 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.878837    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5455c074e8646fc3ec407f079d44b7f59727b3213a39f1d3154425bca2222bb"} err="failed to get container status \"c5455c074e8646fc3ec407f079d44b7f59727b3213a39f1d3154425bca2222bb\": rpc error: code = NotFound desc = could not find container \"c5455c074e8646fc3ec407f079d44b7f59727b3213a39f1d3154425bca2222bb\": container with ID starting with c5455c074e8646fc3ec407f079d44b7f59727b3213a39f1d3154425bca2222bb not found: ID does not exist"
Sep 09 04:11:31.890238 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.890218    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"]
Sep 09 04:11:31.893416 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.893391    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:31.893939 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:31.893922    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-45a82a13-579d99dcdb-7ggrf"]
Sep 09 04:11:32.034624 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.034596    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:32.306392 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.306357    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"]
Sep 09 04:11:32.306794 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.306781    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8" containerName="payload-pre-processing"
Sep 09 04:11:32.306838 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.306796    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8" containerName="payload-pre-processing"
Sep 09 04:11:32.306838 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.306814    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e04c640-f330-4114-a731-e213b88b8e08" containerName="revoke-keys"
Sep 09 04:11:32.306838 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.306819    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e04c640-f330-4114-a731-e213b88b8e08" containerName="revoke-keys"
Sep 09 04:11:32.306930 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.306874    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8" containerName="payload-pre-processing"
Sep 09 04:11:32.306930 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.306884    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e04c640-f330-4114-a731-e213b88b8e08" containerName="revoke-keys"
Sep 09 04:11:32.311152 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.311136    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.313478 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.313435    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-3748ac1b-gw-openshift-default-dockercfg-nkndb\""
Sep 09 04:11:32.320036 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.320017    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"]
Sep 09 04:11:32.330002 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.329976    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:32.410962 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.410921    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a216ae-f782-4e27-a059-68b865e67ab8-istiod-ca-cert\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.411129 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.410972    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-envoy\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.411129 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.411042    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-token\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.411129 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.411091    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-data\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.411129 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.411120    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-workload-certs\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.411356 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.411177    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-podinfo\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.411356 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.411231    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4rrm\" (UniqueName: \"kubernetes.io/projected/f9a216ae-f782-4e27-a059-68b865e67ab8-kube-api-access-d4rrm\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.411484 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.411364    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-workload-socket\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.411484 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.411394    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-credential-socket\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.511878 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.511847    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-data\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.511878 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.511884    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-workload-certs\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.512124 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.511905    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-podinfo\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.512124 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.511933    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4rrm\" (UniqueName: \"kubernetes.io/projected/f9a216ae-f782-4e27-a059-68b865e67ab8-kube-api-access-d4rrm\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.512124 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.511998    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-workload-socket\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.512124 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.512018    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-credential-socket\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.512124 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.512044    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a216ae-f782-4e27-a059-68b865e67ab8-istiod-ca-cert\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.512124 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.512074    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-envoy\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.512124 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.512105    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-token\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.512581 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.512291    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-workload-certs\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.512581 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.512419    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-credential-socket\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.512581 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.512428    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-workload-socket\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.512581 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.512547    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-data\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.513186 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.513166    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a216ae-f782-4e27-a059-68b865e67ab8-istiod-ca-cert\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.514553 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.514523    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-podinfo\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.514688 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.514553    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-envoy\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.519711 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.519682    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-token\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.520107 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.520085    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4rrm\" (UniqueName: \"kubernetes.io/projected/f9a216ae-f782-4e27-a059-68b865e67ab8-kube-api-access-d4rrm\") pod \"e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") " pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.523754 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.523733    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:32.623155 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.623086    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:32.897766 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.897745    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"]
Sep 09 04:11:32.899623 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:11:32.899595    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a216ae_f782_4e27_a059_68b865e67ab8.slice/crio-cd3f9a307c8ca28ed1093d22797e5f2488a6228d0429b31317836a88a432dbd5 WatchSource:0}: Error finding container cd3f9a307c8ca28ed1093d22797e5f2488a6228d0429b31317836a88a432dbd5: Status 404 returned error can't find the container with id cd3f9a307c8ca28ed1093d22797e5f2488a6228d0429b31317836a88a432dbd5
Sep 09 04:11:32.901861 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.901828    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:11:32.901963 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.901910    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:11:32.901963 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.901957    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:11:32.990665 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.990643    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8" path="/var/lib/kubelet/pods/141e64cd-bf2a-4ff1-bfca-fbc5b5792bf8/volumes"
Sep 09 04:11:32.990993 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:32.990968    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e04c640-f330-4114-a731-e213b88b8e08" path="/var/lib/kubelet/pods/5e04c640-f330-4114-a731-e213b88b8e08/volumes"
Sep 09 04:11:33.139081 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:33.139046    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:33.889682 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:33.889650    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4" event={"ID":"f9a216ae-f782-4e27-a059-68b865e67ab8","Type":"ContainerStarted","Data":"9d6dd1dbe5fe1c10f8396cd254879f135949804975d69296be7bafeedbc4d6b6"}
Sep 09 04:11:33.889833 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:33.889731    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4" event={"ID":"f9a216ae-f782-4e27-a059-68b865e67ab8","Type":"ContainerStarted","Data":"cd3f9a307c8ca28ed1093d22797e5f2488a6228d0429b31317836a88a432dbd5"}
Sep 09 04:11:33.915742 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:33.915580    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4" podStartSLOduration=1.91556343 podStartE2EDuration="1.91556343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:11:33.914846562 +0000 UTC m=+793.441217271" watchObservedRunningTime="2026-09-09 04:11:33.91556343 +0000 UTC m=+793.441934140"
Sep 09 04:11:34.624085 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:34.624056    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:34.629078 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:34.629055    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:34.895970 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:34.895901    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:34.896711 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:34.896694    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:35.442596 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:35.442562    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:35.545493 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:35.545466    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:35.642791 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:35.642760    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:35.743528 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:35.743499    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:35.844671 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:35.844649    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:36.129471 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.129380    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:36.230664 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.230640    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:36.241158 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.241130    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"]
Sep 09 04:11:36.244981 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.244966    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"
Sep 09 04:11:36.247179 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.247159    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3748ac1b\""
Sep 09 04:11:36.247179 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.247176    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-3748ac1b-dockercfg-dknqf\""
Sep 09 04:11:36.254663 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.254641    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"]
Sep 09 04:11:36.289935 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.287391    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:36.349132 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.349104    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfefda0-7b12-4044-a711-124f778ca9f3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q\" (UID: \"bcfefda0-7b12-4044-a711-124f778ca9f3\") " pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"
Sep 09 04:11:36.349253 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.349221    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trlw\" (UniqueName: \"kubernetes.io/projected/bcfefda0-7b12-4044-a711-124f778ca9f3-kube-api-access-9trlw\") pod \"payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q\" (UID: \"bcfefda0-7b12-4044-a711-124f778ca9f3\") " pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"
Sep 09 04:11:36.450156 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.450132    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfefda0-7b12-4044-a711-124f778ca9f3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q\" (UID: \"bcfefda0-7b12-4044-a711-124f778ca9f3\") " pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"
Sep 09 04:11:36.450507 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.450229    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trlw\" (UniqueName: \"kubernetes.io/projected/bcfefda0-7b12-4044-a711-124f778ca9f3-kube-api-access-9trlw\") pod \"payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q\" (UID: \"bcfefda0-7b12-4044-a711-124f778ca9f3\") " pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"
Sep 09 04:11:36.450828 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.450807    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfefda0-7b12-4044-a711-124f778ca9f3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q\" (UID: \"bcfefda0-7b12-4044-a711-124f778ca9f3\") " pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"
Sep 09 04:11:36.458614 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.458590    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trlw\" (UniqueName: \"kubernetes.io/projected/bcfefda0-7b12-4044-a711-124f778ca9f3-kube-api-access-9trlw\") pod \"payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q\" (UID: \"bcfefda0-7b12-4044-a711-124f778ca9f3\") " pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"
Sep 09 04:11:36.557490 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.557421    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"
Sep 09 04:11:36.579291 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.579261    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:11:36.652086 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.652062    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-envoy\") pod \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") "
Sep 09 04:11:36.652192 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.652108    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-workload-socket\") pod \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") "
Sep 09 04:11:36.652192 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.652135    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prds7\" (UniqueName: \"kubernetes.io/projected/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-kube-api-access-prds7\") pod \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") "
Sep 09 04:11:36.652280 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.652212    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-workload-certs\") pod \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") "
Sep 09 04:11:36.652280 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.652252    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-podinfo\") pod \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") "
Sep 09 04:11:36.652372 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.652278    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-data\") pod \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") "
Sep 09 04:11:36.652421 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.652382    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istiod-ca-cert\") pod \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") "
Sep 09 04:11:36.652421 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.652410    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" (UID: "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:11:36.652535 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.652458    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-credential-socket\") pod \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") "
Sep 09 04:11:36.652594 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.652544    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-data" (OuterVolumeSpecName: "istio-data") pod "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" (UID: "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:11:36.652594 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.652559    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-token\") pod \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\" (UID: \"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1\") "
Sep 09 04:11:36.652594 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.652561    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" (UID: "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:11:36.652757 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.652718    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" (UID: "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:11:36.652757 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.652751    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" (UID: "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:11:36.653228 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.653128    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-workload-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:11:36.653228 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.653156    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-data\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:11:36.653228 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.653170    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istiod-ca-cert\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:11:36.653228 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.653183    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-credential-socket\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:11:36.653228 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.653198    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-workload-socket\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:11:36.654411 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.654385    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" (UID: "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:11:36.654610 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.654586    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" (UID: "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:11:36.654610 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.654599    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-kube-api-access-prds7" (OuterVolumeSpecName: "kube-api-access-prds7") pod "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" (UID: "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1"). InnerVolumeSpecName "kube-api-access-prds7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:11:36.654790 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.654774    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-token" (OuterVolumeSpecName: "istio-token") pod "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" (UID: "4fce7a56-ef91-4ae4-83f6-f3ec809d55d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:11:36.754425 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.754406    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-podinfo\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:11:36.754425 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.754425    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-token\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:11:36.754558 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.754435    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-istio-envoy\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:11:36.754558 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.754456    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prds7\" (UniqueName: \"kubernetes.io/projected/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1-kube-api-access-prds7\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:11:36.836331 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.836308    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"]
Sep 09 04:11:36.837789 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:11:36.837769    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfefda0_7b12_4044_a711_124f778ca9f3.slice/crio-60b8d777ca7fabc037e013d1bacb7877a941fc72cd57a6ca71899c1fc27948f3 WatchSource:0}: Error finding container 60b8d777ca7fabc037e013d1bacb7877a941fc72cd57a6ca71899c1fc27948f3: Status 404 returned error can't find the container with id 60b8d777ca7fabc037e013d1bacb7877a941fc72cd57a6ca71899c1fc27948f3
Sep 09 04:11:36.908726 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.908703    2553 generic.go:356] "Generic (PLEG): container finished" podID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerID="d3606989e07a5e81cc3ef56cd78ddb893eab9c0248a2dbd9809de5383aafeb4b" exitCode=0
Sep 09 04:11:36.908800 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.908767    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"
Sep 09 04:11:36.908800 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.908780    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk" event={"ID":"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1","Type":"ContainerDied","Data":"d3606989e07a5e81cc3ef56cd78ddb893eab9c0248a2dbd9809de5383aafeb4b"}
Sep 09 04:11:36.908895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.908825    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk" event={"ID":"4fce7a56-ef91-4ae4-83f6-f3ec809d55d1","Type":"ContainerDied","Data":"01e9581615f235c6cf30d23df911fd2aafa2d6ab510798b8775460cfca2de7c2"}
Sep 09 04:11:36.908895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.908845    2553 scope.go:117] "RemoveContainer" containerID="d3606989e07a5e81cc3ef56cd78ddb893eab9c0248a2dbd9809de5383aafeb4b"
Sep 09 04:11:36.912525 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.911689    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q" event={"ID":"bcfefda0-7b12-4044-a711-124f778ca9f3","Type":"ContainerStarted","Data":"60b8d777ca7fabc037e013d1bacb7877a941fc72cd57a6ca71899c1fc27948f3"}
Sep 09 04:11:36.912525 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.911734    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"
Sep 09 04:11:36.919758 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.919320    2553 scope.go:117] "RemoveContainer" containerID="8648d7e8287c2bb1246c7331cd72b00d4b59cfb545f7d1b2600461a8754b50d3"
Sep 09 04:11:36.927879 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.927749    2553 scope.go:117] "RemoveContainer" containerID="d3606989e07a5e81cc3ef56cd78ddb893eab9c0248a2dbd9809de5383aafeb4b"
Sep 09 04:11:36.928040 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:11:36.928020    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3606989e07a5e81cc3ef56cd78ddb893eab9c0248a2dbd9809de5383aafeb4b\": container with ID starting with d3606989e07a5e81cc3ef56cd78ddb893eab9c0248a2dbd9809de5383aafeb4b not found: ID does not exist" containerID="d3606989e07a5e81cc3ef56cd78ddb893eab9c0248a2dbd9809de5383aafeb4b"
Sep 09 04:11:36.928107 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.928046    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3606989e07a5e81cc3ef56cd78ddb893eab9c0248a2dbd9809de5383aafeb4b"} err="failed to get container status \"d3606989e07a5e81cc3ef56cd78ddb893eab9c0248a2dbd9809de5383aafeb4b\": rpc error: code = NotFound desc = could not find container \"d3606989e07a5e81cc3ef56cd78ddb893eab9c0248a2dbd9809de5383aafeb4b\": container with ID starting with d3606989e07a5e81cc3ef56cd78ddb893eab9c0248a2dbd9809de5383aafeb4b not found: ID does not exist"
Sep 09 04:11:36.928107 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.928062    2553 scope.go:117] "RemoveContainer" containerID="8648d7e8287c2bb1246c7331cd72b00d4b59cfb545f7d1b2600461a8754b50d3"
Sep 09 04:11:36.928311 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:11:36.928290    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8648d7e8287c2bb1246c7331cd72b00d4b59cfb545f7d1b2600461a8754b50d3\": container with ID starting with 8648d7e8287c2bb1246c7331cd72b00d4b59cfb545f7d1b2600461a8754b50d3 not found: ID does not exist" containerID="8648d7e8287c2bb1246c7331cd72b00d4b59cfb545f7d1b2600461a8754b50d3"
Sep 09 04:11:36.928400 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.928315    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8648d7e8287c2bb1246c7331cd72b00d4b59cfb545f7d1b2600461a8754b50d3"} err="failed to get container status \"8648d7e8287c2bb1246c7331cd72b00d4b59cfb545f7d1b2600461a8754b50d3\": rpc error: code = NotFound desc = could not find container \"8648d7e8287c2bb1246c7331cd72b00d4b59cfb545f7d1b2600461a8754b50d3\": container with ID starting with 8648d7e8287c2bb1246c7331cd72b00d4b59cfb545f7d1b2600461a8754b50d3 not found: ID does not exist"
Sep 09 04:11:36.938952 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.938898    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q" podStartSLOduration=0.938884767 podStartE2EDuration="938.884767ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:11:36.93743267 +0000 UTC m=+796.463803380" watchObservedRunningTime="2026-09-09 04:11:36.938884767 +0000 UTC m=+796.465255476"
Sep 09 04:11:36.958133 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.958108    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"]
Sep 09 04:11:36.963576 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.963547    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-45a82a13-openshift-default-6787ddc7cd-fkjlk"]
Sep 09 04:11:36.989810 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:36.989785    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" path="/var/lib/kubelet/pods/4fce7a56-ef91-4ae4-83f6-f3ec809d55d1/volumes"
Sep 09 04:11:37.917873 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:37.917846    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q" event={"ID":"bcfefda0-7b12-4044-a711-124f778ca9f3","Type":"ContainerStarted","Data":"af2d4d5823604c7d4c37594f8c43dd376c65f7b29f47451c51d8273cd0ce07a7"}
Sep 09 04:11:43.597225 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:43.597143    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:48.558995 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:48.558959    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:11:48.925330 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:48.925220    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"
Sep 09 04:11:50.520962 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:50.519884    2553 patch_prober.go:28] interesting pod/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": read tcp 10.134.0.2:58478->10.134.0.38:15021: read: connection reset by peer" start-of-body=
Sep 09 04:11:50.520962 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:50.519947    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4" podUID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": read tcp 10.134.0.2:58478->10.134.0.38:15021: read: connection reset by peer"
Sep 09 04:11:50.991361 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:50.991335    2553 generic.go:356] "Generic (PLEG): container finished" podID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerID="9d6dd1dbe5fe1c10f8396cd254879f135949804975d69296be7bafeedbc4d6b6" exitCode=0
Sep 09 04:11:50.991516 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:50.991404    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4" event={"ID":"f9a216ae-f782-4e27-a059-68b865e67ab8","Type":"ContainerDied","Data":"9d6dd1dbe5fe1c10f8396cd254879f135949804975d69296be7bafeedbc4d6b6"}
Sep 09 04:11:50.991789 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:50.991771    2553 scope.go:117] "RemoveContainer" containerID="9d6dd1dbe5fe1c10f8396cd254879f135949804975d69296be7bafeedbc4d6b6"
Sep 09 04:11:50.992898 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:50.992872    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:11:50.992968 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:50.992938    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:11:50.993026 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:50.992970    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:11:52.001656 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:52.001610    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4" event={"ID":"f9a216ae-f782-4e27-a059-68b865e67ab8","Type":"ContainerStarted","Data":"d4531877703d29e337584aa0a85147c463620b53fd9a91f022437b6cc9b77e55"}
Sep 09 04:11:52.623295 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:52.623260    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:52.623514 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:52.623491    2553 patch_prober.go:28] interesting pod/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 04:11:52.623576 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:52.623540    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4" podUID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 04:11:53.149834 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.149801    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"]
Sep 09 04:11:53.150243 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.150231    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerName="istio-proxy"
Sep 09 04:11:53.150243 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.150244    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerName="istio-proxy"
Sep 09 04:11:53.150330 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.150257    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerName="istio-proxy"
Sep 09 04:11:53.150330 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.150263    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerName="istio-proxy"
Sep 09 04:11:53.150330 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.150317    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerName="istio-proxy"
Sep 09 04:11:53.150330 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.150324    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fce7a56-ef91-4ae4-83f6-f3ec809d55d1" containerName="istio-proxy"
Sep 09 04:11:53.156384 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.156364    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.158673 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.158647    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-85732c\"/\"kube-root-ca.crt\""
Sep 09 04:11:53.159747 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.159711    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-85732c\"/\"openshift-service-ca.crt\""
Sep 09 04:11:53.159873 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.159811    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-85732c\"/\"simulator-w4-85732c-kserve-self-signed-certs\""
Sep 09 04:11:53.159972 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.159950    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-85732c\"/\"default-dockercfg-scmzk\""
Sep 09 04:11:53.163466 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.163424    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"]
Sep 09 04:11:53.262570 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.262544    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:11:53.320070 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.320027    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-tmp-dir\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.320246 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.320083    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55596728-74e8-4c54-ba37-f930bf20e6d0-tls-certs\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.320246 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.320156    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkj62\" (UniqueName: \"kubernetes.io/projected/55596728-74e8-4c54-ba37-f930bf20e6d0-kube-api-access-rkj62\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.320246 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.320217    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-home\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.320246 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.320235    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-dshm\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.320520 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.320264    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-model-cache\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.421819 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.421740    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55596728-74e8-4c54-ba37-f930bf20e6d0-tls-certs\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.421819 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.421800    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkj62\" (UniqueName: \"kubernetes.io/projected/55596728-74e8-4c54-ba37-f930bf20e6d0-kube-api-access-rkj62\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.422024 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.421879    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-home\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.422024 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.421905    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-dshm\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.422024 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.421931    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-model-cache\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.422024 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.422000    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-tmp-dir\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.422389 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.422361    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-home\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.422512 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.422406    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-model-cache\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.422512 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.422427    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-tmp-dir\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.424491 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.424468    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55596728-74e8-4c54-ba37-f930bf20e6d0-tls-certs\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.424643 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.424627    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-dshm\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.429837 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.429811    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkj62\" (UniqueName: \"kubernetes.io/projected/55596728-74e8-4c54-ba37-f930bf20e6d0-kube-api-access-rkj62\") pod \"simulator-w4-85732c-kserve-6f95b54b95-kx42p\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") " pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.468388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.468346    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:11:53.624366 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.624328    2553 patch_prober.go:28] interesting pod/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 04:11:53.624527 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.624392    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4" podUID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 04:11:53.760571 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:53.760545    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"]
Sep 09 04:11:53.762766 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:11:53.762739    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55596728_74e8_4c54_ba37_f930bf20e6d0.slice/crio-8d18c5707332526755bd626267ecb95ee28bebd44f10b66e86d39229333eec7f WatchSource:0}: Error finding container 8d18c5707332526755bd626267ecb95ee28bebd44f10b66e86d39229333eec7f: Status 404 returned error can't find the container with id 8d18c5707332526755bd626267ecb95ee28bebd44f10b66e86d39229333eec7f
Sep 09 04:11:54.020151 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:54.020038    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p" event={"ID":"55596728-74e8-4c54-ba37-f930bf20e6d0","Type":"ContainerStarted","Data":"9fec177b66a229490df6616767b73ca58e96f949c976a5ce52db00100772911a"}
Sep 09 04:11:54.020151 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:54.020109    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p" event={"ID":"55596728-74e8-4c54-ba37-f930bf20e6d0","Type":"ContainerStarted","Data":"8d18c5707332526755bd626267ecb95ee28bebd44f10b66e86d39229333eec7f"}
Sep 09 04:11:54.037840 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:54.037793    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p" podStartSLOduration=1.037778641 podStartE2EDuration="1.037778641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:11:54.03596272 +0000 UTC m=+813.562333428" watchObservedRunningTime="2026-09-09 04:11:54.037778641 +0000 UTC m=+813.564149412"
Sep 09 04:11:54.624331 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:54.624293    2553 patch_prober.go:28] interesting pod/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 04:11:54.624784 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:54.624363    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4" podUID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 04:11:55.623739 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:55.623700    2553 patch_prober.go:28] interesting pod/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 04:11:55.623923 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:55.623758    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4" podUID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 04:11:56.624001 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:56.623967    2553 patch_prober.go:28] interesting pod/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 04:11:56.624379 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:56.624021    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4" podUID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 04:11:57.627537 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:57.627508    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:57.627962 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:57.627783    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:11:57.628431 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:11:57.628413    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:12:00.427339 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:00.427305    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76"]
Sep 09 04:12:00.432943 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:00.432906    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76"
Sep 09 04:12:00.436322 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:00.436301    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:12:00.436505 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:00.436331    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:12:00.438412 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:00.438392    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76"]
Sep 09 04:12:00.590406 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:00.590379    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1e7513-26d5-4e32-aaad-d970603557ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76\" (UID: \"7b1e7513-26d5-4e32-aaad-d970603557ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76"
Sep 09 04:12:00.691305 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:00.691246    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1e7513-26d5-4e32-aaad-d970603557ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76\" (UID: \"7b1e7513-26d5-4e32-aaad-d970603557ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76"
Sep 09 04:12:00.691823 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:00.691806    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1e7513-26d5-4e32-aaad-d970603557ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76\" (UID: \"7b1e7513-26d5-4e32-aaad-d970603557ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76"
Sep 09 04:12:00.745367 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:00.745348    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76"
Sep 09 04:12:01.017411 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:01.017387    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76"]
Sep 09 04:12:01.019016 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:12:01.018990    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1e7513_26d5_4e32_aaad_d970603557ab.slice/crio-e6b3f05155d074fd651e04c522c5a4a07315ffe154c5bf08a8d9970732ccb3a5 WatchSource:0}: Error finding container e6b3f05155d074fd651e04c522c5a4a07315ffe154c5bf08a8d9970732ccb3a5: Status 404 returned error can't find the container with id e6b3f05155d074fd651e04c522c5a4a07315ffe154c5bf08a8d9970732ccb3a5
Sep 09 04:12:01.059969 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:01.059937    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76" event={"ID":"7b1e7513-26d5-4e32-aaad-d970603557ab","Type":"ContainerStarted","Data":"e6b3f05155d074fd651e04c522c5a4a07315ffe154c5bf08a8d9970732ccb3a5"}
Sep 09 04:12:02.066435 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:02.066401    2553 generic.go:356] "Generic (PLEG): container finished" podID="7b1e7513-26d5-4e32-aaad-d970603557ab" containerID="7f588ca5cce3fe6e134c975a06771ccf00a7724f356a78ca34e75541940d0d64" exitCode=0
Sep 09 04:12:02.066787 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:02.066485    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76" event={"ID":"7b1e7513-26d5-4e32-aaad-d970603557ab","Type":"ContainerDied","Data":"7f588ca5cce3fe6e134c975a06771ccf00a7724f356a78ca34e75541940d0d64"}
Sep 09 04:12:03.205180 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:03.205159    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76"
Sep 09 04:12:03.313061 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:03.313035    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1e7513-26d5-4e32-aaad-d970603557ab-maas-api-service-ca\") pod \"7b1e7513-26d5-4e32-aaad-d970603557ab\" (UID: \"7b1e7513-26d5-4e32-aaad-d970603557ab\") "
Sep 09 04:12:03.313358 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:03.313338    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1e7513-26d5-4e32-aaad-d970603557ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b1e7513-26d5-4e32-aaad-d970603557ab" (UID: "7b1e7513-26d5-4e32-aaad-d970603557ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:12:03.414308 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:03.414251    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1e7513-26d5-4e32-aaad-d970603557ab-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:03.468914 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:03.468897    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:12:03.469012 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:03.468932    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:12:03.481263 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:03.481232    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:12:04.077732 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:04.077706    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76"
Sep 09 04:12:04.077899 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:04.077741    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76" event={"ID":"7b1e7513-26d5-4e32-aaad-d970603557ab","Type":"ContainerDied","Data":"e6b3f05155d074fd651e04c522c5a4a07315ffe154c5bf08a8d9970732ccb3a5"}
Sep 09 04:12:04.077899 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:04.077784    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6b3f05155d074fd651e04c522c5a4a07315ffe154c5bf08a8d9970732ccb3a5"
Sep 09 04:12:04.089168 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:04.089136    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:12:05.759367 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:05.759332    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:12:06.898772 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:06.898739    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:06.933008 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:06.932971    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"]
Sep 09 04:12:06.933262 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:06.933224    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q" podUID="bcfefda0-7b12-4044-a711-124f778ca9f3" containerName="payload-processing" containerID="cri-o://af2d4d5823604c7d4c37594f8c43dd376c65f7b29f47451c51d8273cd0ce07a7" gracePeriod=30
Sep 09 04:12:06.999512 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:06.999478    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:07.097471 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.097409    2553 generic.go:356] "Generic (PLEG): container finished" podID="bcfefda0-7b12-4044-a711-124f778ca9f3" containerID="af2d4d5823604c7d4c37594f8c43dd376c65f7b29f47451c51d8273cd0ce07a7" exitCode=0
Sep 09 04:12:07.097655 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.097489    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q" event={"ID":"bcfefda0-7b12-4044-a711-124f778ca9f3","Type":"ContainerDied","Data":"af2d4d5823604c7d4c37594f8c43dd376c65f7b29f47451c51d8273cd0ce07a7"}
Sep 09 04:12:07.103422 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.103378    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76"]
Sep 09 04:12:07.104990 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.104965    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:07.108017 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.107991    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3748ac1b-q7p76"]
Sep 09 04:12:07.159668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.159561    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:07.192303 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.192284    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"
Sep 09 04:12:07.237465 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.237144    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"]
Sep 09 04:12:07.237465 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.237400    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4" podUID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerName="istio-proxy" containerID="cri-o://d4531877703d29e337584aa0a85147c463620b53fd9a91f022437b6cc9b77e55" gracePeriod=30
Sep 09 04:12:07.256095 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.256005    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:07.352472 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.351902    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9trlw\" (UniqueName: \"kubernetes.io/projected/bcfefda0-7b12-4044-a711-124f778ca9f3-kube-api-access-9trlw\") pod \"bcfefda0-7b12-4044-a711-124f778ca9f3\" (UID: \"bcfefda0-7b12-4044-a711-124f778ca9f3\") "
Sep 09 04:12:07.352472 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.352180    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfefda0-7b12-4044-a711-124f778ca9f3-plugins-config-volume\") pod \"bcfefda0-7b12-4044-a711-124f778ca9f3\" (UID: \"bcfefda0-7b12-4044-a711-124f778ca9f3\") "
Sep 09 04:12:07.356468 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.353073    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcfefda0-7b12-4044-a711-124f778ca9f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bcfefda0-7b12-4044-a711-124f778ca9f3" (UID: "bcfefda0-7b12-4044-a711-124f778ca9f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:12:07.356468 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.353569    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfefda0-7b12-4044-a711-124f778ca9f3-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:07.356468 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.355188    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcfefda0-7b12-4044-a711-124f778ca9f3-kube-api-access-9trlw" (OuterVolumeSpecName: "kube-api-access-9trlw") pod "bcfefda0-7b12-4044-a711-124f778ca9f3" (UID: "bcfefda0-7b12-4044-a711-124f778ca9f3"). InnerVolumeSpecName "kube-api-access-9trlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:12:07.454625 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.454556    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9trlw\" (UniqueName: \"kubernetes.io/projected/bcfefda0-7b12-4044-a711-124f778ca9f3-kube-api-access-9trlw\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:07.728247 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.728216    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:07.997893 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.997810    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"]
Sep 09 04:12:07.998324 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.998271    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1e7513-26d5-4e32-aaad-d970603557ab" containerName="revoke-keys"
Sep 09 04:12:07.998324 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.998284    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1e7513-26d5-4e32-aaad-d970603557ab" containerName="revoke-keys"
Sep 09 04:12:07.998324 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.998302    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcfefda0-7b12-4044-a711-124f778ca9f3" containerName="payload-processing"
Sep 09 04:12:07.998324 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.998308    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcfefda0-7b12-4044-a711-124f778ca9f3" containerName="payload-processing"
Sep 09 04:12:07.998505 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.998371    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1e7513-26d5-4e32-aaad-d970603557ab" containerName="revoke-keys"
Sep 09 04:12:07.998505 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:07.998396    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcfefda0-7b12-4044-a711-124f778ca9f3" containerName="payload-processing"
Sep 09 04:12:08.001739 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.001725    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.004036 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.004017    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3dbd54be-openshift-default-dockercfg-68thg\""
Sep 09 04:12:08.011811 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.011784    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"]
Sep 09 04:12:08.023337 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.023316    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:08.059305 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.059272    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec65c3d3-5523-490f-8705-38285c3e8925-istio-podinfo\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.059407 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.059325    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-workload-certs\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.059407 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.059360    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz44z\" (UniqueName: \"kubernetes.io/projected/ec65c3d3-5523-490f-8705-38285c3e8925-kube-api-access-gz44z\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.059407 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.059397    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec65c3d3-5523-490f-8705-38285c3e8925-istiod-ca-cert\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.059621 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.059417    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-istio-data\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.059621 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.059530    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-workload-socket\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.059621 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.059566    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-credential-socket\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.059767 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.059625    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec65c3d3-5523-490f-8705-38285c3e8925-istio-token\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.059767 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.059660    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-istio-envoy\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.106690 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.106656    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q" event={"ID":"bcfefda0-7b12-4044-a711-124f778ca9f3","Type":"ContainerDied","Data":"60b8d777ca7fabc037e013d1bacb7877a941fc72cd57a6ca71899c1fc27948f3"}
Sep 09 04:12:08.106876 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.106728    2553 scope.go:117] "RemoveContainer" containerID="af2d4d5823604c7d4c37594f8c43dd376c65f7b29f47451c51d8273cd0ce07a7"
Sep 09 04:12:08.106944 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.106897    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"
Sep 09 04:12:08.132579 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.132553    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"]
Sep 09 04:12:08.136835 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.136816    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3748ac1b-c74c49cd-dzm6q"]
Sep 09 04:12:08.160725 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.160700    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-istio-envoy\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.160814 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.160746    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec65c3d3-5523-490f-8705-38285c3e8925-istio-podinfo\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.160814 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.160793    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-workload-certs\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.160894 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.160818    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz44z\" (UniqueName: \"kubernetes.io/projected/ec65c3d3-5523-490f-8705-38285c3e8925-kube-api-access-gz44z\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.160894 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.160857    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec65c3d3-5523-490f-8705-38285c3e8925-istiod-ca-cert\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.160894 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.160888    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-istio-data\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.161044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.160923    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-workload-socket\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.161044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.160949    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-credential-socket\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.161044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.161007    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec65c3d3-5523-490f-8705-38285c3e8925-istio-token\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.161214 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.161187    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-workload-certs\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.161291 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.161228    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-istio-data\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.161367 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.161313    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-workload-socket\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.161464 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.161421    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-credential-socket\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.161648 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.161629    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec65c3d3-5523-490f-8705-38285c3e8925-istiod-ca-cert\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.163362 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.163336    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-istio-envoy\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.163472 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.163341    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec65c3d3-5523-490f-8705-38285c3e8925-istio-podinfo\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.168695 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.168666    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec65c3d3-5523-490f-8705-38285c3e8925-istio-token\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.168695 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.168670    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz44z\" (UniqueName: \"kubernetes.io/projected/ec65c3d3-5523-490f-8705-38285c3e8925-kube-api-access-gz44z\") pod \"e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") " pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.313203 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.313128    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:08.315517 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.315495    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:08.588543 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.588523    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"]
Sep 09 04:12:08.590278 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:12:08.590255    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec65c3d3_5523_490f_8705_38285c3e8925.slice/crio-0eaed17a223351fef21321a1dac43c58c94e886e68478047f93b81f4a24bcc15 WatchSource:0}: Error finding container 0eaed17a223351fef21321a1dac43c58c94e886e68478047f93b81f4a24bcc15: Status 404 returned error can't find the container with id 0eaed17a223351fef21321a1dac43c58c94e886e68478047f93b81f4a24bcc15
Sep 09 04:12:08.592740 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.592701    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:12:08.592812 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.592794    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:12:08.592861 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.592842    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:12:08.989553 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.989518    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b1e7513-26d5-4e32-aaad-d970603557ab" path="/var/lib/kubelet/pods/7b1e7513-26d5-4e32-aaad-d970603557ab/volumes"
Sep 09 04:12:08.989920 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:08.989906    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcfefda0-7b12-4044-a711-124f778ca9f3" path="/var/lib/kubelet/pods/bcfefda0-7b12-4044-a711-124f778ca9f3/volumes"
Sep 09 04:12:09.113152 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:09.113120    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9" event={"ID":"ec65c3d3-5523-490f-8705-38285c3e8925","Type":"ContainerStarted","Data":"e7e60a870e2354b2e1330485bb7e2fdf8094ca1e2a6152f2407900921134a282"}
Sep 09 04:12:09.113572 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:09.113185    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9" event={"ID":"ec65c3d3-5523-490f-8705-38285c3e8925","Type":"ContainerStarted","Data":"0eaed17a223351fef21321a1dac43c58c94e886e68478047f93b81f4a24bcc15"}
Sep 09 04:12:09.133945 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:09.133903    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9" podStartSLOduration=2.133890745 podStartE2EDuration="2.133890745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:12:09.131899404 +0000 UTC m=+828.658270138" watchObservedRunningTime="2026-09-09 04:12:09.133890745 +0000 UTC m=+828.660261450"
Sep 09 04:12:09.314346 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:09.314258    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:09.318929 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:09.318908    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:10.119498 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:10.119473    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:10.120319 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:10.120298    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:10.460827 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:10.460749    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:10.553294 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:10.553262    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:11.606828 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:11.606795    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:11.707508 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:11.707471    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:11.807240 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:11.807211    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:12.210893 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.210862    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"]
Sep 09 04:12:12.215337 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.215309    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"
Sep 09 04:12:12.217667 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.217648    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3dbd54be\""
Sep 09 04:12:12.224646 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.224624    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"]
Sep 09 04:12:12.306619 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.306593    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413c8b1d-48b4-4d32-b044-153861b018ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf\" (UID: \"413c8b1d-48b4-4d32-b044-153861b018ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"
Sep 09 04:12:12.306715 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.306627    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z5tc\" (UniqueName: \"kubernetes.io/projected/413c8b1d-48b4-4d32-b044-153861b018ec-kube-api-access-4z5tc\") pod \"payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf\" (UID: \"413c8b1d-48b4-4d32-b044-153861b018ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"
Sep 09 04:12:12.329793 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.329763    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:12.407597 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.407563    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413c8b1d-48b4-4d32-b044-153861b018ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf\" (UID: \"413c8b1d-48b4-4d32-b044-153861b018ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"
Sep 09 04:12:12.407748 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.407602    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z5tc\" (UniqueName: \"kubernetes.io/projected/413c8b1d-48b4-4d32-b044-153861b018ec-kube-api-access-4z5tc\") pod \"payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf\" (UID: \"413c8b1d-48b4-4d32-b044-153861b018ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"
Sep 09 04:12:12.408322 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.408301    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413c8b1d-48b4-4d32-b044-153861b018ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf\" (UID: \"413c8b1d-48b4-4d32-b044-153861b018ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"
Sep 09 04:12:12.416339 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.416316    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z5tc\" (UniqueName: \"kubernetes.io/projected/413c8b1d-48b4-4d32-b044-153861b018ec-kube-api-access-4z5tc\") pod \"payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf\" (UID: \"413c8b1d-48b4-4d32-b044-153861b018ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"
Sep 09 04:12:12.429620 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.429572    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:12.490101 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.490080    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:12:12.509044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.508274    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-workload-socket\") pod \"f9a216ae-f782-4e27-a059-68b865e67ab8\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") "
Sep 09 04:12:12.509044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.508368    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-token\") pod \"f9a216ae-f782-4e27-a059-68b865e67ab8\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") "
Sep 09 04:12:12.509044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.508407    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4rrm\" (UniqueName: \"kubernetes.io/projected/f9a216ae-f782-4e27-a059-68b865e67ab8-kube-api-access-d4rrm\") pod \"f9a216ae-f782-4e27-a059-68b865e67ab8\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") "
Sep 09 04:12:12.509044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.508488    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a216ae-f782-4e27-a059-68b865e67ab8-istiod-ca-cert\") pod \"f9a216ae-f782-4e27-a059-68b865e67ab8\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") "
Sep 09 04:12:12.509044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.508521    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-envoy\") pod \"f9a216ae-f782-4e27-a059-68b865e67ab8\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") "
Sep 09 04:12:12.509044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.508543    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-data\") pod \"f9a216ae-f782-4e27-a059-68b865e67ab8\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") "
Sep 09 04:12:12.509044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.508596    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-workload-certs\") pod \"f9a216ae-f782-4e27-a059-68b865e67ab8\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") "
Sep 09 04:12:12.509044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.508624    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-podinfo\") pod \"f9a216ae-f782-4e27-a059-68b865e67ab8\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") "
Sep 09 04:12:12.509044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.508654    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-credential-socket\") pod \"f9a216ae-f782-4e27-a059-68b865e67ab8\" (UID: \"f9a216ae-f782-4e27-a059-68b865e67ab8\") "
Sep 09 04:12:12.509696 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.509333    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f9a216ae-f782-4e27-a059-68b865e67ab8" (UID: "f9a216ae-f782-4e27-a059-68b865e67ab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:12:12.509696 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.509575    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f9a216ae-f782-4e27-a059-68b865e67ab8" (UID: "f9a216ae-f782-4e27-a059-68b865e67ab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:12:12.510503 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.510479    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-data" (OuterVolumeSpecName: "istio-data") pod "f9a216ae-f782-4e27-a059-68b865e67ab8" (UID: "f9a216ae-f782-4e27-a059-68b865e67ab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:12:12.510964 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.510941    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f9a216ae-f782-4e27-a059-68b865e67ab8" (UID: "f9a216ae-f782-4e27-a059-68b865e67ab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:12:12.511102 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.511085    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a216ae-f782-4e27-a059-68b865e67ab8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f9a216ae-f782-4e27-a059-68b865e67ab8" (UID: "f9a216ae-f782-4e27-a059-68b865e67ab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:12:12.512547 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.512521    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a216ae-f782-4e27-a059-68b865e67ab8-kube-api-access-d4rrm" (OuterVolumeSpecName: "kube-api-access-d4rrm") pod "f9a216ae-f782-4e27-a059-68b865e67ab8" (UID: "f9a216ae-f782-4e27-a059-68b865e67ab8"). InnerVolumeSpecName "kube-api-access-d4rrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:12:12.512720 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.512694    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-token" (OuterVolumeSpecName: "istio-token") pod "f9a216ae-f782-4e27-a059-68b865e67ab8" (UID: "f9a216ae-f782-4e27-a059-68b865e67ab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:12:12.513176 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.513151    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f9a216ae-f782-4e27-a059-68b865e67ab8" (UID: "f9a216ae-f782-4e27-a059-68b865e67ab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:12:12.513290 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.513238    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f9a216ae-f782-4e27-a059-68b865e67ab8" (UID: "f9a216ae-f782-4e27-a059-68b865e67ab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:12:12.527459 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.527430    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"
Sep 09 04:12:12.609843 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.609815    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-workload-socket\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:12.609843 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.609843    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-token\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:12.610212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.609857    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4rrm\" (UniqueName: \"kubernetes.io/projected/f9a216ae-f782-4e27-a059-68b865e67ab8-kube-api-access-d4rrm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:12.610212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.609868    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a216ae-f782-4e27-a059-68b865e67ab8-istiod-ca-cert\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:12.610212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.609886    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-envoy\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:12.610212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.609907    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-data\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:12.610212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.609920    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-workload-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:12.610212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.609932    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a216ae-f782-4e27-a059-68b865e67ab8-istio-podinfo\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:12.610212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.609944    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a216ae-f782-4e27-a059-68b865e67ab8-credential-socket\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:12.796805 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:12.796778    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"]
Sep 09 04:12:12.798189 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:12:12.798167    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413c8b1d_48b4_4d32_b044_153861b018ec.slice/crio-215ea43d7922eb9287825d51341cdd250eebb1aa811ffc489fa02ea13c1cd313 WatchSource:0}: Error finding container 215ea43d7922eb9287825d51341cdd250eebb1aa811ffc489fa02ea13c1cd313: Status 404 returned error can't find the container with id 215ea43d7922eb9287825d51341cdd250eebb1aa811ffc489fa02ea13c1cd313
Sep 09 04:12:13.138048 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.137962    2553 generic.go:356] "Generic (PLEG): container finished" podID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerID="d4531877703d29e337584aa0a85147c463620b53fd9a91f022437b6cc9b77e55" exitCode=0
Sep 09 04:12:13.138048 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.138031    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"
Sep 09 04:12:13.138256 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.138050    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4" event={"ID":"f9a216ae-f782-4e27-a059-68b865e67ab8","Type":"ContainerDied","Data":"d4531877703d29e337584aa0a85147c463620b53fd9a91f022437b6cc9b77e55"}
Sep 09 04:12:13.138256 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.138112    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4" event={"ID":"f9a216ae-f782-4e27-a059-68b865e67ab8","Type":"ContainerDied","Data":"cd3f9a307c8ca28ed1093d22797e5f2488a6228d0429b31317836a88a432dbd5"}
Sep 09 04:12:13.138256 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.138142    2553 scope.go:117] "RemoveContainer" containerID="d4531877703d29e337584aa0a85147c463620b53fd9a91f022437b6cc9b77e55"
Sep 09 04:12:13.139807 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.139775    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf" event={"ID":"413c8b1d-48b4-4d32-b044-153861b018ec","Type":"ContainerStarted","Data":"e8f7c18e939793a0db7c6d764a0a63dfd5f02b07c301cbc4fb9bb495cf6c4f76"}
Sep 09 04:12:13.139932 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.139822    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf" event={"ID":"413c8b1d-48b4-4d32-b044-153861b018ec","Type":"ContainerStarted","Data":"215ea43d7922eb9287825d51341cdd250eebb1aa811ffc489fa02ea13c1cd313"}
Sep 09 04:12:13.139932 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.139846    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"
Sep 09 04:12:13.147658 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.147645    2553 scope.go:117] "RemoveContainer" containerID="9d6dd1dbe5fe1c10f8396cd254879f135949804975d69296be7bafeedbc4d6b6"
Sep 09 04:12:13.154561 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.154542    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"]
Sep 09 04:12:13.156970 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.156952    2553 scope.go:117] "RemoveContainer" containerID="d4531877703d29e337584aa0a85147c463620b53fd9a91f022437b6cc9b77e55"
Sep 09 04:12:13.157265 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:12:13.157234    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4531877703d29e337584aa0a85147c463620b53fd9a91f022437b6cc9b77e55\": container with ID starting with d4531877703d29e337584aa0a85147c463620b53fd9a91f022437b6cc9b77e55 not found: ID does not exist" containerID="d4531877703d29e337584aa0a85147c463620b53fd9a91f022437b6cc9b77e55"
Sep 09 04:12:13.157370 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.157274    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4531877703d29e337584aa0a85147c463620b53fd9a91f022437b6cc9b77e55"} err="failed to get container status \"d4531877703d29e337584aa0a85147c463620b53fd9a91f022437b6cc9b77e55\": rpc error: code = NotFound desc = could not find container \"d4531877703d29e337584aa0a85147c463620b53fd9a91f022437b6cc9b77e55\": container with ID starting with d4531877703d29e337584aa0a85147c463620b53fd9a91f022437b6cc9b77e55 not found: ID does not exist"
Sep 09 04:12:13.157370 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.157294    2553 scope.go:117] "RemoveContainer" containerID="9d6dd1dbe5fe1c10f8396cd254879f135949804975d69296be7bafeedbc4d6b6"
Sep 09 04:12:13.157628 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:12:13.157610    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6dd1dbe5fe1c10f8396cd254879f135949804975d69296be7bafeedbc4d6b6\": container with ID starting with 9d6dd1dbe5fe1c10f8396cd254879f135949804975d69296be7bafeedbc4d6b6 not found: ID does not exist" containerID="9d6dd1dbe5fe1c10f8396cd254879f135949804975d69296be7bafeedbc4d6b6"
Sep 09 04:12:13.157693 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.157635    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6dd1dbe5fe1c10f8396cd254879f135949804975d69296be7bafeedbc4d6b6"} err="failed to get container status \"9d6dd1dbe5fe1c10f8396cd254879f135949804975d69296be7bafeedbc4d6b6\": rpc error: code = NotFound desc = could not find container \"9d6dd1dbe5fe1c10f8396cd254879f135949804975d69296be7bafeedbc4d6b6\": container with ID starting with 9d6dd1dbe5fe1c10f8396cd254879f135949804975d69296be7bafeedbc4d6b6 not found: ID does not exist"
Sep 09 04:12:13.158957 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.158934    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-3748ac1b-gw-openshift-default-546f8b59cb-n8hq4"]
Sep 09 04:12:13.172268 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:13.172230    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf" podStartSLOduration=1.172218004 podStartE2EDuration="1.172218004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:12:13.170517801 +0000 UTC m=+832.696888511" watchObservedRunningTime="2026-09-09 04:12:13.172218004 +0000 UTC m=+832.698588712"
Sep 09 04:12:14.989107 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:14.989076    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9a216ae-f782-4e27-a059-68b865e67ab8" path="/var/lib/kubelet/pods/f9a216ae-f782-4e27-a059-68b865e67ab8/volumes"
Sep 09 04:12:21.903375 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:21.903340    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:22.769436 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:22.769403    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:12:24.149535 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:24.149509    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"
Sep 09 04:12:32.517805 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:32.517764    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:33.261754 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:33.261721    2553 generic.go:356] "Generic (PLEG): container finished" podID="ec65c3d3-5523-490f-8705-38285c3e8925" containerID="e7e60a870e2354b2e1330485bb7e2fdf8094ca1e2a6152f2407900921134a282" exitCode=0
Sep 09 04:12:33.261947 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:33.261790    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9" event={"ID":"ec65c3d3-5523-490f-8705-38285c3e8925","Type":"ContainerDied","Data":"e7e60a870e2354b2e1330485bb7e2fdf8094ca1e2a6152f2407900921134a282"}
Sep 09 04:12:33.262178 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:33.262164    2553 scope.go:117] "RemoveContainer" containerID="e7e60a870e2354b2e1330485bb7e2fdf8094ca1e2a6152f2407900921134a282"
Sep 09 04:12:33.263264 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:33.263237    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:12:33.263365 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:33.263318    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:12:33.263427 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:33.263364    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:12:34.269316 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:34.269265    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9" event={"ID":"ec65c3d3-5523-490f-8705-38285c3e8925","Type":"ContainerStarted","Data":"304742ada6f53fcd202807eb3cf0d50061e83053c220062f965532512bb0cb38"}
Sep 09 04:12:34.314381 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:34.314342    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:34.314789 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:34.314759    2553 patch_prober.go:28] interesting pod/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 04:12:34.314909 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:34.314822    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9" podUID="ec65c3d3-5523-490f-8705-38285c3e8925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 04:12:35.313565 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:35.313526    2553 patch_prober.go:28] interesting pod/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 04:12:35.314058 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:35.313604    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9" podUID="ec65c3d3-5523-490f-8705-38285c3e8925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 04:12:36.290209 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.286993    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r"]
Sep 09 04:12:36.290209 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.288007    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerName="istio-proxy"
Sep 09 04:12:36.290209 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.288028    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerName="istio-proxy"
Sep 09 04:12:36.290209 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.288250    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerName="istio-proxy"
Sep 09 04:12:36.290209 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.288270    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerName="istio-proxy"
Sep 09 04:12:36.291748 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.291719    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r"
Sep 09 04:12:36.294382 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.294346    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:12:36.294785 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.294766    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:12:36.298594 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.297338    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r"]
Sep 09 04:12:36.325522 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.317802    2553 patch_prober.go:28] interesting pod/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 04:12:36.325522 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.317863    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9" podUID="ec65c3d3-5523-490f-8705-38285c3e8925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 04:12:36.447828 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.447796    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69fcf4d-9dd5-493f-959b-8129e4e33ddd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r\" (UID: \"c69fcf4d-9dd5-493f-959b-8129e4e33ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r"
Sep 09 04:12:36.549171 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.549084    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69fcf4d-9dd5-493f-959b-8129e4e33ddd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r\" (UID: \"c69fcf4d-9dd5-493f-959b-8129e4e33ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r"
Sep 09 04:12:36.549728 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.549709    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69fcf4d-9dd5-493f-959b-8129e4e33ddd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r\" (UID: \"c69fcf4d-9dd5-493f-959b-8129e4e33ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r"
Sep 09 04:12:36.628916 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.628886    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r"
Sep 09 04:12:36.900522 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:36.900418    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r"]
Sep 09 04:12:36.904938 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:12:36.904905    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69fcf4d_9dd5_493f_959b_8129e4e33ddd.slice/crio-a126713828ee97c402b23408904621aaca8ee2516e2ad6b3a0503957845c980b WatchSource:0}: Error finding container a126713828ee97c402b23408904621aaca8ee2516e2ad6b3a0503957845c980b: Status 404 returned error can't find the container with id a126713828ee97c402b23408904621aaca8ee2516e2ad6b3a0503957845c980b
Sep 09 04:12:37.285287 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:37.285238    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r" event={"ID":"c69fcf4d-9dd5-493f-959b-8129e4e33ddd","Type":"ContainerStarted","Data":"15dcf9faac24b926a42f711121cc12a5c8b873bb1bd5ceeb9d44143f4c095fcd"}
Sep 09 04:12:37.285430 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:37.285329    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r" event={"ID":"c69fcf4d-9dd5-493f-959b-8129e4e33ddd","Type":"ContainerStarted","Data":"a126713828ee97c402b23408904621aaca8ee2516e2ad6b3a0503957845c980b"}
Sep 09 04:12:37.314100 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:37.314060    2553 patch_prober.go:28] interesting pod/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 04:12:37.314264 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:37.314121    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9" podUID="ec65c3d3-5523-490f-8705-38285c3e8925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 04:12:37.761690 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:37.761638    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r" podStartSLOduration=1.761621596 podStartE2EDuration="1.761621596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:12:37.306573992 +0000 UTC m=+856.832944702" watchObservedRunningTime="2026-09-09 04:12:37.761621596 +0000 UTC m=+857.287992308"
Sep 09 04:12:37.762545 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:37.762523    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:12:38.292215 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:38.292168    2553 generic.go:356] "Generic (PLEG): container finished" podID="c69fcf4d-9dd5-493f-959b-8129e4e33ddd" containerID="15dcf9faac24b926a42f711121cc12a5c8b873bb1bd5ceeb9d44143f4c095fcd" exitCode=0
Sep 09 04:12:38.292359 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:38.292227    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r" event={"ID":"c69fcf4d-9dd5-493f-959b-8129e4e33ddd","Type":"ContainerDied","Data":"15dcf9faac24b926a42f711121cc12a5c8b873bb1bd5ceeb9d44143f4c095fcd"}
Sep 09 04:12:38.314011 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:38.313974    2553 patch_prober.go:28] interesting pod/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 04:12:38.314186 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:38.314045    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9" podUID="ec65c3d3-5523-490f-8705-38285c3e8925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 04:12:39.317845 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.317774    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:39.318363 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.318118    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:39.319006 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.318985    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:39.347437 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.347409    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"]
Sep 09 04:12:39.347926 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.347909    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerName="istio-proxy"
Sep 09 04:12:39.348017 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.347929    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a216ae-f782-4e27-a059-68b865e67ab8" containerName="istio-proxy"
Sep 09 04:12:39.350642 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.350624    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.353323 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.353300    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4ad5fa\"/\"openshift-service-ca.crt\""
Sep 09 04:12:39.353769 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.353305    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4ad5fa\"/\"kube-root-ca.crt\""
Sep 09 04:12:39.353769 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.353374    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4ad5fa\"/\"autores-model-4ad5fa-kserve-self-signed-certs\""
Sep 09 04:12:39.353769 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.353381    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4ad5fa\"/\"default-dockercfg-98xrt\""
Sep 09 04:12:39.360068 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.360046    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"]
Sep 09 04:12:39.373652 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.373629    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:39.439344 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.439322    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r"
Sep 09 04:12:39.479215 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.479192    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23d766d4-c9a9-46ea-859e-dda2ac75d307-tls-certs\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.479338 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.479226    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-tmp-dir\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.479338 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.479257    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-home\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.479501 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.479476    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-dshm\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.479572 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.479539    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-model-cache\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.479648 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.479579    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2pr\" (UniqueName: \"kubernetes.io/projected/23d766d4-c9a9-46ea-859e-dda2ac75d307-kube-api-access-xk2pr\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.580820 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.580756    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69fcf4d-9dd5-493f-959b-8129e4e33ddd-maas-api-service-ca\") pod \"c69fcf4d-9dd5-493f-959b-8129e4e33ddd\" (UID: \"c69fcf4d-9dd5-493f-959b-8129e4e33ddd\") "
Sep 09 04:12:39.580973 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.580957    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-dshm\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.581044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.580989    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-model-cache\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.581044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.581012    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2pr\" (UniqueName: \"kubernetes.io/projected/23d766d4-c9a9-46ea-859e-dda2ac75d307-kube-api-access-xk2pr\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.581156 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.581078    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23d766d4-c9a9-46ea-859e-dda2ac75d307-tls-certs\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.581156 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.581098    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69fcf4d-9dd5-493f-959b-8129e4e33ddd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c69fcf4d-9dd5-493f-959b-8129e4e33ddd" (UID: "c69fcf4d-9dd5-493f-959b-8129e4e33ddd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:12:39.581156 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.581108    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-tmp-dir\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.581319 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.581180    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-home\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.581386 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.581333    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69fcf4d-9dd5-493f-959b-8129e4e33ddd-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:39.581386 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.581379    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-model-cache\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.581564 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.581545    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-tmp-dir\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.581564 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.581560    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-home\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.583380 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.583366    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-dshm\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.583649 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.583632    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23d766d4-c9a9-46ea-859e-dda2ac75d307-tls-certs\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.589229 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.589213    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2pr\" (UniqueName: \"kubernetes.io/projected/23d766d4-c9a9-46ea-859e-dda2ac75d307-kube-api-access-xk2pr\") pod \"autores-model-4ad5fa-kserve-6567875ccf-wv2kn\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") " pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.664357 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.664320    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:39.936428 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:39.936402    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"]
Sep 09 04:12:39.938348 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:12:39.938327    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23d766d4_c9a9_46ea_859e_dda2ac75d307.slice/crio-543ee2a535172891823cef911c91b935b04e1e434ceb7901cab5fc3128b55e30 WatchSource:0}: Error finding container 543ee2a535172891823cef911c91b935b04e1e434ceb7901cab5fc3128b55e30: Status 404 returned error can't find the container with id 543ee2a535172891823cef911c91b935b04e1e434ceb7901cab5fc3128b55e30
Sep 09 04:12:40.311059 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:40.311019    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn" event={"ID":"23d766d4-c9a9-46ea-859e-dda2ac75d307","Type":"ContainerStarted","Data":"f67d76ccb38c197f117a0992df3133616480f74998779cbf779e77526208ba62"}
Sep 09 04:12:40.311236 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:40.311069    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn" event={"ID":"23d766d4-c9a9-46ea-859e-dda2ac75d307","Type":"ContainerStarted","Data":"543ee2a535172891823cef911c91b935b04e1e434ceb7901cab5fc3128b55e30"}
Sep 09 04:12:40.312263 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:40.312238    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r"
Sep 09 04:12:40.312380 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:40.312240    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r" event={"ID":"c69fcf4d-9dd5-493f-959b-8129e4e33ddd","Type":"ContainerDied","Data":"a126713828ee97c402b23408904621aaca8ee2516e2ad6b3a0503957845c980b"}
Sep 09 04:12:40.312429 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:40.312418    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a126713828ee97c402b23408904621aaca8ee2516e2ad6b3a0503957845c980b"
Sep 09 04:12:40.328877 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:40.328839    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn" podStartSLOduration=1.328827011 podStartE2EDuration="1.328827011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:12:40.327351191 +0000 UTC m=+859.853721918" watchObservedRunningTime="2026-09-09 04:12:40.328827011 +0000 UTC m=+859.855197718"
Sep 09 04:12:42.149641 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.149611    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:42.191851 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.191820    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"]
Sep 09 04:12:42.192091 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.192062    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf" podUID="413c8b1d-48b4-4d32-b044-153861b018ec" containerName="payload-pre-processing" containerID="cri-o://e8f7c18e939793a0db7c6d764a0a63dfd5f02b07c301cbc4fb9bb495cf6c4f76" gracePeriod=30
Sep 09 04:12:42.219840 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.219818    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:42.320438 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.320387    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:42.325521 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.325499    2553 generic.go:356] "Generic (PLEG): container finished" podID="413c8b1d-48b4-4d32-b044-153861b018ec" containerID="e8f7c18e939793a0db7c6d764a0a63dfd5f02b07c301cbc4fb9bb495cf6c4f76" exitCode=0
Sep 09 04:12:42.325637 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.325575    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf" event={"ID":"413c8b1d-48b4-4d32-b044-153861b018ec","Type":"ContainerDied","Data":"e8f7c18e939793a0db7c6d764a0a63dfd5f02b07c301cbc4fb9bb495cf6c4f76"}
Sep 09 04:12:42.355965 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.355930    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r"]
Sep 09 04:12:42.359523 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.359498    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3dbd54be-qnd8r"]
Sep 09 04:12:42.376165 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.376070    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:42.449741 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.449717    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"
Sep 09 04:12:42.511227 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.511201    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z5tc\" (UniqueName: \"kubernetes.io/projected/413c8b1d-48b4-4d32-b044-153861b018ec-kube-api-access-4z5tc\") pod \"413c8b1d-48b4-4d32-b044-153861b018ec\" (UID: \"413c8b1d-48b4-4d32-b044-153861b018ec\") "
Sep 09 04:12:42.511385 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.511325    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413c8b1d-48b4-4d32-b044-153861b018ec-plugins-config-volume\") pod \"413c8b1d-48b4-4d32-b044-153861b018ec\" (UID: \"413c8b1d-48b4-4d32-b044-153861b018ec\") "
Sep 09 04:12:42.511678 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.511657    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413c8b1d-48b4-4d32-b044-153861b018ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "413c8b1d-48b4-4d32-b044-153861b018ec" (UID: "413c8b1d-48b4-4d32-b044-153861b018ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:12:42.513399 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.513377    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413c8b1d-48b4-4d32-b044-153861b018ec-kube-api-access-4z5tc" (OuterVolumeSpecName: "kube-api-access-4z5tc") pod "413c8b1d-48b4-4d32-b044-153861b018ec" (UID: "413c8b1d-48b4-4d32-b044-153861b018ec"). InnerVolumeSpecName "kube-api-access-4z5tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:12:42.613111 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.613075    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413c8b1d-48b4-4d32-b044-153861b018ec-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:42.613111 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.613106    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z5tc\" (UniqueName: \"kubernetes.io/projected/413c8b1d-48b4-4d32-b044-153861b018ec-kube-api-access-4z5tc\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:42.997403 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:42.997372    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c69fcf4d-9dd5-493f-959b-8129e4e33ddd" path="/var/lib/kubelet/pods/c69fcf4d-9dd5-493f-959b-8129e4e33ddd/volumes"
Sep 09 04:12:43.331752 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:43.331681    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf" event={"ID":"413c8b1d-48b4-4d32-b044-153861b018ec","Type":"ContainerDied","Data":"215ea43d7922eb9287825d51341cdd250eebb1aa811ffc489fa02ea13c1cd313"}
Sep 09 04:12:43.331752 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:43.331710    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"
Sep 09 04:12:43.331752 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:43.331736    2553 scope.go:117] "RemoveContainer" containerID="e8f7c18e939793a0db7c6d764a0a63dfd5f02b07c301cbc4fb9bb495cf6c4f76"
Sep 09 04:12:43.348680 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:43.348660    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"]
Sep 09 04:12:43.354431 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:43.354412    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3dbd54be-7b6f8b7dcd-j9lcf"]
Sep 09 04:12:43.537853 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:43.537828    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"]
Sep 09 04:12:43.538032 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:43.538012    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9" podUID="ec65c3d3-5523-490f-8705-38285c3e8925" containerName="istio-proxy" containerID="cri-o://304742ada6f53fcd202807eb3cf0d50061e83053c220062f965532512bb0cb38" gracePeriod=30
Sep 09 04:12:43.550702 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:43.550680    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:44.006324 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:44.006288    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:44.311406 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:44.311322    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:44.776161 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:44.776135    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:44.988653 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:44.988619    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="413c8b1d-48b4-4d32-b044-153861b018ec" path="/var/lib/kubelet/pods/413c8b1d-48b4-4d32-b044-153861b018ec/volumes"
Sep 09 04:12:46.865868 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:46.865828    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:46.966879 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:46.966846    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:48.113222 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.113194    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:48.212423 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.212390    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:48.312695 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.312664    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:48.622585 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.622561    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:48.720552 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.720525    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:48.733577 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.733545    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"]
Sep 09 04:12:48.734702 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.734676    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69fcf4d-9dd5-493f-959b-8129e4e33ddd" containerName="revoke-keys"
Sep 09 04:12:48.734702 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.734703    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69fcf4d-9dd5-493f-959b-8129e4e33ddd" containerName="revoke-keys"
Sep 09 04:12:48.734883 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.734758    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="413c8b1d-48b4-4d32-b044-153861b018ec" containerName="payload-pre-processing"
Sep 09 04:12:48.734883 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.734767    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="413c8b1d-48b4-4d32-b044-153861b018ec" containerName="payload-pre-processing"
Sep 09 04:12:48.735015 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.735002    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69fcf4d-9dd5-493f-959b-8129e4e33ddd" containerName="revoke-keys"
Sep 09 04:12:48.735101 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.735092    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="413c8b1d-48b4-4d32-b044-153861b018ec" containerName="payload-pre-processing"
Sep 09 04:12:48.741432 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.740641    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"
Sep 09 04:12:48.743935 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.743844    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-1be20a5f\""
Sep 09 04:12:48.744073 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.743934    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"]
Sep 09 04:12:48.761880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.761859    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"]
Sep 09 04:12:48.765859 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.765843    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"
Sep 09 04:12:48.768186 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.768163    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-1be20a5f-dockercfg-8zvpv\""
Sep 09 04:12:48.773294 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.772040    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a53460-d7e4-4e29-9c31-fa7b4102a8a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl\" (UID: \"c0a53460-d7e4-4e29-9c31-fa7b4102a8a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"
Sep 09 04:12:48.773294 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.772195    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg2qm\" (UniqueName: \"kubernetes.io/projected/c0a53460-d7e4-4e29-9c31-fa7b4102a8a9-kube-api-access-xg2qm\") pod \"payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl\" (UID: \"c0a53460-d7e4-4e29-9c31-fa7b4102a8a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"
Sep 09 04:12:48.775035 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.775014    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"]
Sep 09 04:12:48.799179 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.799137    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:48.873055 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873026    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-workload-certs\") pod \"ec65c3d3-5523-490f-8705-38285c3e8925\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") "
Sep 09 04:12:48.873055 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873061    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-workload-socket\") pod \"ec65c3d3-5523-490f-8705-38285c3e8925\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") "
Sep 09 04:12:48.873315 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873092    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-credential-socket\") pod \"ec65c3d3-5523-490f-8705-38285c3e8925\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") "
Sep 09 04:12:48.873315 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873132    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec65c3d3-5523-490f-8705-38285c3e8925-istiod-ca-cert\") pod \"ec65c3d3-5523-490f-8705-38285c3e8925\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") "
Sep 09 04:12:48.873315 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873162    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec65c3d3-5523-490f-8705-38285c3e8925-istio-token\") pod \"ec65c3d3-5523-490f-8705-38285c3e8925\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") "
Sep 09 04:12:48.873315 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873187    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-istio-envoy\") pod \"ec65c3d3-5523-490f-8705-38285c3e8925\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") "
Sep 09 04:12:48.873315 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873238    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec65c3d3-5523-490f-8705-38285c3e8925-istio-podinfo\") pod \"ec65c3d3-5523-490f-8705-38285c3e8925\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") "
Sep 09 04:12:48.873315 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873299    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec65c3d3-5523-490f-8705-38285c3e8925" (UID: "ec65c3d3-5523-490f-8705-38285c3e8925"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:12:48.873627 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873331    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz44z\" (UniqueName: \"kubernetes.io/projected/ec65c3d3-5523-490f-8705-38285c3e8925-kube-api-access-gz44z\") pod \"ec65c3d3-5523-490f-8705-38285c3e8925\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") "
Sep 09 04:12:48.873627 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873364    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-istio-data\") pod \"ec65c3d3-5523-490f-8705-38285c3e8925\" (UID: \"ec65c3d3-5523-490f-8705-38285c3e8925\") "
Sep 09 04:12:48.873627 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873383    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec65c3d3-5523-490f-8705-38285c3e8925" (UID: "ec65c3d3-5523-490f-8705-38285c3e8925"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:12:48.873627 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873464    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec65c3d3-5523-490f-8705-38285c3e8925" (UID: "ec65c3d3-5523-490f-8705-38285c3e8925"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:12:48.873627 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873585    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg2qm\" (UniqueName: \"kubernetes.io/projected/c0a53460-d7e4-4e29-9c31-fa7b4102a8a9-kube-api-access-xg2qm\") pod \"payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl\" (UID: \"c0a53460-d7e4-4e29-9c31-fa7b4102a8a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"
Sep 09 04:12:48.873860 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873674    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec65c3d3-5523-490f-8705-38285c3e8925" (UID: "ec65c3d3-5523-490f-8705-38285c3e8925"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:12:48.873860 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873744    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6f5cc3-045c-4601-b99a-5493fc0774d1-plugins-config-volume\") pod \"payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct\" (UID: \"7a6f5cc3-045c-4601-b99a-5493fc0774d1\") " pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"
Sep 09 04:12:48.873860 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873795    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a53460-d7e4-4e29-9c31-fa7b4102a8a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl\" (UID: \"c0a53460-d7e4-4e29-9c31-fa7b4102a8a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"
Sep 09 04:12:48.874007 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873879    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgkt9\" (UniqueName: \"kubernetes.io/projected/7a6f5cc3-045c-4601-b99a-5493fc0774d1-kube-api-access-bgkt9\") pod \"payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct\" (UID: \"7a6f5cc3-045c-4601-b99a-5493fc0774d1\") " pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"
Sep 09 04:12:48.874007 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.873878    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec65c3d3-5523-490f-8705-38285c3e8925-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec65c3d3-5523-490f-8705-38285c3e8925" (UID: "ec65c3d3-5523-490f-8705-38285c3e8925"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:12:48.874175 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.874123    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-istio-data\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:48.874175 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.874151    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-workload-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:48.874175 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.874163    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-workload-socket\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:48.874175 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.874172    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-credential-socket\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:48.874514 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.874185    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec65c3d3-5523-490f-8705-38285c3e8925-istiod-ca-cert\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:48.874886 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.874860    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a53460-d7e4-4e29-9c31-fa7b4102a8a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl\" (UID: \"c0a53460-d7e4-4e29-9c31-fa7b4102a8a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"
Sep 09 04:12:48.875837 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.875816    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec65c3d3-5523-490f-8705-38285c3e8925-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec65c3d3-5523-490f-8705-38285c3e8925" (UID: "ec65c3d3-5523-490f-8705-38285c3e8925"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:12:48.875929 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.875844    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec65c3d3-5523-490f-8705-38285c3e8925" (UID: "ec65c3d3-5523-490f-8705-38285c3e8925"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:12:48.876100 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.876081    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec65c3d3-5523-490f-8705-38285c3e8925-kube-api-access-gz44z" (OuterVolumeSpecName: "kube-api-access-gz44z") pod "ec65c3d3-5523-490f-8705-38285c3e8925" (UID: "ec65c3d3-5523-490f-8705-38285c3e8925"). InnerVolumeSpecName "kube-api-access-gz44z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:12:48.876100 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.876092    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec65c3d3-5523-490f-8705-38285c3e8925-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec65c3d3-5523-490f-8705-38285c3e8925" (UID: "ec65c3d3-5523-490f-8705-38285c3e8925"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:12:48.884659 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.884640    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg2qm\" (UniqueName: \"kubernetes.io/projected/c0a53460-d7e4-4e29-9c31-fa7b4102a8a9-kube-api-access-xg2qm\") pod \"payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl\" (UID: \"c0a53460-d7e4-4e29-9c31-fa7b4102a8a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"
Sep 09 04:12:48.975015 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.974988    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkt9\" (UniqueName: \"kubernetes.io/projected/7a6f5cc3-045c-4601-b99a-5493fc0774d1-kube-api-access-bgkt9\") pod \"payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct\" (UID: \"7a6f5cc3-045c-4601-b99a-5493fc0774d1\") " pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"
Sep 09 04:12:48.975123 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.975109    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6f5cc3-045c-4601-b99a-5493fc0774d1-plugins-config-volume\") pod \"payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct\" (UID: \"7a6f5cc3-045c-4601-b99a-5493fc0774d1\") " pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"
Sep 09 04:12:48.975190 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.975177    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec65c3d3-5523-490f-8705-38285c3e8925-istio-podinfo\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:48.975228 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.975196    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz44z\" (UniqueName: \"kubernetes.io/projected/ec65c3d3-5523-490f-8705-38285c3e8925-kube-api-access-gz44z\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:48.975228 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.975210    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec65c3d3-5523-490f-8705-38285c3e8925-istio-token\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:48.975228 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.975222    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec65c3d3-5523-490f-8705-38285c3e8925-istio-envoy\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:12:48.975715 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.975697    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6f5cc3-045c-4601-b99a-5493fc0774d1-plugins-config-volume\") pod \"payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct\" (UID: \"7a6f5cc3-045c-4601-b99a-5493fc0774d1\") " pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"
Sep 09 04:12:48.982898 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:48.982875    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgkt9\" (UniqueName: \"kubernetes.io/projected/7a6f5cc3-045c-4601-b99a-5493fc0774d1-kube-api-access-bgkt9\") pod \"payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct\" (UID: \"7a6f5cc3-045c-4601-b99a-5493fc0774d1\") " pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"
Sep 09 04:12:49.054369 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.054318    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"
Sep 09 04:12:49.076662 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.076642    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"
Sep 09 04:12:49.327644 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.327621    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"]
Sep 09 04:12:49.328826 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:12:49.328805    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a53460_d7e4_4e29_9c31_fa7b4102a8a9.slice/crio-49f1fe7c4fd39ad1643806d2915c09fb3991cbc1aaa2253aab282ab4f28aa978 WatchSource:0}: Error finding container 49f1fe7c4fd39ad1643806d2915c09fb3991cbc1aaa2253aab282ab4f28aa978: Status 404 returned error can't find the container with id 49f1fe7c4fd39ad1643806d2915c09fb3991cbc1aaa2253aab282ab4f28aa978
Sep 09 04:12:49.355715 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.355696    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"]
Sep 09 04:12:49.359482 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:12:49.359432    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6f5cc3_045c_4601_b99a_5493fc0774d1.slice/crio-a1f05e719e7ab23d6feb785af1c89ee837d07d87971bc1379683a7b083c2f25f WatchSource:0}: Error finding container a1f05e719e7ab23d6feb785af1c89ee837d07d87971bc1379683a7b083c2f25f: Status 404 returned error can't find the container with id a1f05e719e7ab23d6feb785af1c89ee837d07d87971bc1379683a7b083c2f25f
Sep 09 04:12:49.366281 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.366259    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl" event={"ID":"c0a53460-d7e4-4e29-9c31-fa7b4102a8a9","Type":"ContainerStarted","Data":"49f1fe7c4fd39ad1643806d2915c09fb3991cbc1aaa2253aab282ab4f28aa978"}
Sep 09 04:12:49.368083 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.368060    2553 generic.go:356] "Generic (PLEG): container finished" podID="ec65c3d3-5523-490f-8705-38285c3e8925" containerID="304742ada6f53fcd202807eb3cf0d50061e83053c220062f965532512bb0cb38" exitCode=0
Sep 09 04:12:49.368178 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.368119    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"
Sep 09 04:12:49.368178 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.368133    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9" event={"ID":"ec65c3d3-5523-490f-8705-38285c3e8925","Type":"ContainerDied","Data":"304742ada6f53fcd202807eb3cf0d50061e83053c220062f965532512bb0cb38"}
Sep 09 04:12:49.368290 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.368180    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9" event={"ID":"ec65c3d3-5523-490f-8705-38285c3e8925","Type":"ContainerDied","Data":"0eaed17a223351fef21321a1dac43c58c94e886e68478047f93b81f4a24bcc15"}
Sep 09 04:12:49.368290 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.368210    2553 scope.go:117] "RemoveContainer" containerID="304742ada6f53fcd202807eb3cf0d50061e83053c220062f965532512bb0cb38"
Sep 09 04:12:49.385115 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.385097    2553 scope.go:117] "RemoveContainer" containerID="e7e60a870e2354b2e1330485bb7e2fdf8094ca1e2a6152f2407900921134a282"
Sep 09 04:12:49.393193 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.393161    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"]
Sep 09 04:12:49.395484 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.395464    2553 scope.go:117] "RemoveContainer" containerID="304742ada6f53fcd202807eb3cf0d50061e83053c220062f965532512bb0cb38"
Sep 09 04:12:49.395757 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:12:49.395735    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304742ada6f53fcd202807eb3cf0d50061e83053c220062f965532512bb0cb38\": container with ID starting with 304742ada6f53fcd202807eb3cf0d50061e83053c220062f965532512bb0cb38 not found: ID does not exist" containerID="304742ada6f53fcd202807eb3cf0d50061e83053c220062f965532512bb0cb38"
Sep 09 04:12:49.395856 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.395768    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304742ada6f53fcd202807eb3cf0d50061e83053c220062f965532512bb0cb38"} err="failed to get container status \"304742ada6f53fcd202807eb3cf0d50061e83053c220062f965532512bb0cb38\": rpc error: code = NotFound desc = could not find container \"304742ada6f53fcd202807eb3cf0d50061e83053c220062f965532512bb0cb38\": container with ID starting with 304742ada6f53fcd202807eb3cf0d50061e83053c220062f965532512bb0cb38 not found: ID does not exist"
Sep 09 04:12:49.395856 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.395789    2553 scope.go:117] "RemoveContainer" containerID="e7e60a870e2354b2e1330485bb7e2fdf8094ca1e2a6152f2407900921134a282"
Sep 09 04:12:49.396179 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:12:49.396057    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e60a870e2354b2e1330485bb7e2fdf8094ca1e2a6152f2407900921134a282\": container with ID starting with e7e60a870e2354b2e1330485bb7e2fdf8094ca1e2a6152f2407900921134a282 not found: ID does not exist" containerID="e7e60a870e2354b2e1330485bb7e2fdf8094ca1e2a6152f2407900921134a282"
Sep 09 04:12:49.396179 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.396085    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e60a870e2354b2e1330485bb7e2fdf8094ca1e2a6152f2407900921134a282"} err="failed to get container status \"e7e60a870e2354b2e1330485bb7e2fdf8094ca1e2a6152f2407900921134a282\": rpc error: code = NotFound desc = could not find container \"e7e60a870e2354b2e1330485bb7e2fdf8094ca1e2a6152f2407900921134a282\": container with ID starting with e7e60a870e2354b2e1330485bb7e2fdf8094ca1e2a6152f2407900921134a282 not found: ID does not exist"
Sep 09 04:12:49.397371 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.397348    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3dbd54be-openshift-default-6d6d674d86-ntdz9"]
Sep 09 04:12:49.664679 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.664612    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:49.664679 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.664653    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:49.676741 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:49.676713    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:50.375555 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:50.375523    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct" event={"ID":"7a6f5cc3-045c-4601-b99a-5493fc0774d1","Type":"ContainerStarted","Data":"ecd679f382ebd968f77ddbce289bd04deb8cccd49912b7a6c97bf8230e499aa2"}
Sep 09 04:12:50.376006 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:50.375577    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct" event={"ID":"7a6f5cc3-045c-4601-b99a-5493fc0774d1","Type":"ContainerStarted","Data":"a1f05e719e7ab23d6feb785af1c89ee837d07d87971bc1379683a7b083c2f25f"}
Sep 09 04:12:50.376006 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:50.375608    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"
Sep 09 04:12:50.378135 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:50.378108    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl" event={"ID":"c0a53460-d7e4-4e29-9c31-fa7b4102a8a9","Type":"ContainerStarted","Data":"fbcaf548596656488d5a8aa16bee212b02c251b12ad845df6777a3162423dcc2"}
Sep 09 04:12:50.389400 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:50.389370    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:12:50.391503 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:50.391465    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct" podStartSLOduration=2.391436593 podStartE2EDuration="2.391436593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:12:50.390435556 +0000 UTC m=+869.916806266" watchObservedRunningTime="2026-09-09 04:12:50.391436593 +0000 UTC m=+869.917807302"
Sep 09 04:12:50.422892 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:50.422854    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl" podStartSLOduration=2.42284159 podStartE2EDuration="2.42284159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:12:50.421223349 +0000 UTC m=+869.947594060" watchObservedRunningTime="2026-09-09 04:12:50.42284159 +0000 UTC m=+869.949212300"
Sep 09 04:12:50.990694 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:50.990661    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec65c3d3-5523-490f-8705-38285c3e8925" path="/var/lib/kubelet/pods/ec65c3d3-5523-490f-8705-38285c3e8925/volumes"
Sep 09 04:12:51.383329 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:51.383252    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"
Sep 09 04:12:53.387820 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:53.387787    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:53.437716 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:53.437689    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:12:56.870362 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:12:56.870329    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:13:01.384317 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:01.384286    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"
Sep 09 04:13:02.391481 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:02.391435    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"
Sep 09 04:13:06.429293 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:06.429253    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:10.275538 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.275508    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"]
Sep 09 04:13:10.275955 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.275940    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec65c3d3-5523-490f-8705-38285c3e8925" containerName="istio-proxy"
Sep 09 04:13:10.276054 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.275958    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec65c3d3-5523-490f-8705-38285c3e8925" containerName="istio-proxy"
Sep 09 04:13:10.276054 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.276049    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec65c3d3-5523-490f-8705-38285c3e8925" containerName="istio-proxy"
Sep 09 04:13:10.276131 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.276057    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec65c3d3-5523-490f-8705-38285c3e8925" containerName="istio-proxy"
Sep 09 04:13:10.280712 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.280693    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.283192 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.283171    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-359adf\"/\"kube-root-ca.crt\""
Sep 09 04:13:10.284404 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.284382    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-359adf\"/\"test-model-359adf-kserve-self-signed-certs\""
Sep 09 04:13:10.284503 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.284408    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-359adf\"/\"openshift-service-ca.crt\""
Sep 09 04:13:10.284503 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.284385    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-359adf\"/\"default-dockercfg-wd2z2\""
Sep 09 04:13:10.288669 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.288648    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"]
Sep 09 04:13:10.302297 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.302273    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:10.361753 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.361734    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf792\" (UniqueName: \"kubernetes.io/projected/e162efbb-6638-4b6a-a051-cc6c95e626ca-kube-api-access-pf792\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.361853 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.361765    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-home\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.361853 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.361805    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-model-cache\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.361853 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.361841    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-tmp-dir\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.361981 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.361915    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e162efbb-6638-4b6a-a051-cc6c95e626ca-tls-certs\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.361981 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.361956    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-dshm\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.462884 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.462848    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf792\" (UniqueName: \"kubernetes.io/projected/e162efbb-6638-4b6a-a051-cc6c95e626ca-kube-api-access-pf792\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.463076 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.462900    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-home\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.463076 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.462960    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-model-cache\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.463076 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.462983    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-tmp-dir\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.463076 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.463013    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e162efbb-6638-4b6a-a051-cc6c95e626ca-tls-certs\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.463424 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.463399    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-home\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.463576 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.463428    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-tmp-dir\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.463576 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.463485    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-dshm\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.463576 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.463543    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-model-cache\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.465722 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.465693    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-dshm\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.465907 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.465892    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e162efbb-6638-4b6a-a051-cc6c95e626ca-tls-certs\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.470642 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.470624    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf792\" (UniqueName: \"kubernetes.io/projected/e162efbb-6638-4b6a-a051-cc6c95e626ca-kube-api-access-pf792\") pod \"test-model-359adf-kserve-6dc674c4f7-6g5z2\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") " pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.619222 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.619167    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:10.889907 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:10.889882    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"]
Sep 09 04:13:10.891708 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:13:10.891683    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode162efbb_6638_4b6a_a051_cc6c95e626ca.slice/crio-c636691f6bdacdeba1854e8520432f72ed7c9a8f6c297c85fa4bd2d54be40205 WatchSource:0}: Error finding container c636691f6bdacdeba1854e8520432f72ed7c9a8f6c297c85fa4bd2d54be40205: Status 404 returned error can't find the container with id c636691f6bdacdeba1854e8520432f72ed7c9a8f6c297c85fa4bd2d54be40205
Sep 09 04:13:11.358757 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.358732    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm"]
Sep 09 04:13:11.359166 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.359153    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec65c3d3-5523-490f-8705-38285c3e8925" containerName="istio-proxy"
Sep 09 04:13:11.359210 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.359168    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec65c3d3-5523-490f-8705-38285c3e8925" containerName="istio-proxy"
Sep 09 04:13:11.362239 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.362222    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm"
Sep 09 04:13:11.364639 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.364620    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:13:11.364639 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.364631    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:13:11.370092 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.370068    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm"]
Sep 09 04:13:11.473656 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.473625    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d9d7b07-acc4-44c1-9da7-62cd0a39f928-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm\" (UID: \"6d9d7b07-acc4-44c1-9da7-62cd0a39f928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm"
Sep 09 04:13:11.493253 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.493225    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2" event={"ID":"e162efbb-6638-4b6a-a051-cc6c95e626ca","Type":"ContainerStarted","Data":"4b9e307b798c74bea46aa84d2933147f2bc44f0ca48bd559be7947ceaeaea857"}
Sep 09 04:13:11.493365 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.493265    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2" event={"ID":"e162efbb-6638-4b6a-a051-cc6c95e626ca","Type":"ContainerStarted","Data":"c636691f6bdacdeba1854e8520432f72ed7c9a8f6c297c85fa4bd2d54be40205"}
Sep 09 04:13:11.499706 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.499685    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:11.509854 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.509816    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2" podStartSLOduration=1.50980446 podStartE2EDuration="1.50980446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:13:11.508738143 +0000 UTC m=+891.035108856" watchObservedRunningTime="2026-09-09 04:13:11.50980446 +0000 UTC m=+891.036175170"
Sep 09 04:13:11.574722 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.574702    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d9d7b07-acc4-44c1-9da7-62cd0a39f928-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm\" (UID: \"6d9d7b07-acc4-44c1-9da7-62cd0a39f928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm"
Sep 09 04:13:11.575298 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.575277    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d9d7b07-acc4-44c1-9da7-62cd0a39f928-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm\" (UID: \"6d9d7b07-acc4-44c1-9da7-62cd0a39f928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm"
Sep 09 04:13:11.674479 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.674399    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm"
Sep 09 04:13:11.759377 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.759348    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv"]
Sep 09 04:13:11.764493 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.764472    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv"
Sep 09 04:13:11.771103 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.771028    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv"]
Sep 09 04:13:11.877138 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.877113    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83afef74-0242-4436-9d06-0dca98d8d5f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv\" (UID: \"83afef74-0242-4436-9d06-0dca98d8d5f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv"
Sep 09 04:13:11.946467 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.946413    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm"]
Sep 09 04:13:11.948755 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:13:11.948734    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9d7b07_acc4_44c1_9da7_62cd0a39f928.slice/crio-593934b16c7f9b9558103592d694c16beff7d092682f02bab0e3a3f13865c3c6 WatchSource:0}: Error finding container 593934b16c7f9b9558103592d694c16beff7d092682f02bab0e3a3f13865c3c6: Status 404 returned error can't find the container with id 593934b16c7f9b9558103592d694c16beff7d092682f02bab0e3a3f13865c3c6
Sep 09 04:13:11.977995 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.977970    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83afef74-0242-4436-9d06-0dca98d8d5f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv\" (UID: \"83afef74-0242-4436-9d06-0dca98d8d5f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv"
Sep 09 04:13:11.978508 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:11.978490    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83afef74-0242-4436-9d06-0dca98d8d5f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv\" (UID: \"83afef74-0242-4436-9d06-0dca98d8d5f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv"
Sep 09 04:13:12.078333 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:12.078312    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv"
Sep 09 04:13:12.349342 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:12.349300    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv"]
Sep 09 04:13:12.351211 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:13:12.351187    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83afef74_0242_4436_9d06_0dca98d8d5f3.slice/crio-6f7989b89c1843de6f076742737a26929960fdc942db5ec7033feee20d9ec647 WatchSource:0}: Error finding container 6f7989b89c1843de6f076742737a26929960fdc942db5ec7033feee20d9ec647: Status 404 returned error can't find the container with id 6f7989b89c1843de6f076742737a26929960fdc942db5ec7033feee20d9ec647
Sep 09 04:13:12.502953 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:12.502908    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv" event={"ID":"83afef74-0242-4436-9d06-0dca98d8d5f3","Type":"ContainerStarted","Data":"6f7989b89c1843de6f076742737a26929960fdc942db5ec7033feee20d9ec647"}
Sep 09 04:13:12.504521 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:12.504489    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm" event={"ID":"6d9d7b07-acc4-44c1-9da7-62cd0a39f928","Type":"ContainerStarted","Data":"a9bcea6f3354987f4a98036d8ca73341462c1629a4da7202e5070b8cbf57104a"}
Sep 09 04:13:12.504653 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:12.504547    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm" event={"ID":"6d9d7b07-acc4-44c1-9da7-62cd0a39f928","Type":"ContainerStarted","Data":"593934b16c7f9b9558103592d694c16beff7d092682f02bab0e3a3f13865c3c6"}
Sep 09 04:13:12.521820 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:12.521777    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm" podStartSLOduration=1.5217618000000002 podStartE2EDuration="1.5217618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:13:12.521292498 +0000 UTC m=+892.047663189" watchObservedRunningTime="2026-09-09 04:13:12.5217618 +0000 UTC m=+892.048132509"
Sep 09 04:13:13.511657 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:13.511625    2553 generic.go:356] "Generic (PLEG): container finished" podID="83afef74-0242-4436-9d06-0dca98d8d5f3" containerID="72480dd580506a12143d4172ecea54617a7be62c40ca7e32ebcebf7d52512293" exitCode=0
Sep 09 04:13:13.512109 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:13.511713    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv" event={"ID":"83afef74-0242-4436-9d06-0dca98d8d5f3","Type":"ContainerDied","Data":"72480dd580506a12143d4172ecea54617a7be62c40ca7e32ebcebf7d52512293"}
Sep 09 04:13:13.513254 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:13.513233    2553 generic.go:356] "Generic (PLEG): container finished" podID="6d9d7b07-acc4-44c1-9da7-62cd0a39f928" containerID="a9bcea6f3354987f4a98036d8ca73341462c1629a4da7202e5070b8cbf57104a" exitCode=0
Sep 09 04:13:13.513392 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:13.513316    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm" event={"ID":"6d9d7b07-acc4-44c1-9da7-62cd0a39f928","Type":"ContainerDied","Data":"a9bcea6f3354987f4a98036d8ca73341462c1629a4da7202e5070b8cbf57104a"}
Sep 09 04:13:13.623315 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:13.623287    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:14.646666 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:14.646644    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv"
Sep 09 04:13:14.685766 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:14.685746    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm"
Sep 09 04:13:14.804797 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:14.804728    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d9d7b07-acc4-44c1-9da7-62cd0a39f928-maas-api-service-ca\") pod \"6d9d7b07-acc4-44c1-9da7-62cd0a39f928\" (UID: \"6d9d7b07-acc4-44c1-9da7-62cd0a39f928\") "
Sep 09 04:13:14.804797 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:14.804767    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83afef74-0242-4436-9d06-0dca98d8d5f3-maas-api-service-ca\") pod \"83afef74-0242-4436-9d06-0dca98d8d5f3\" (UID: \"83afef74-0242-4436-9d06-0dca98d8d5f3\") "
Sep 09 04:13:14.805096 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:14.805030    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9d7b07-acc4-44c1-9da7-62cd0a39f928-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d9d7b07-acc4-44c1-9da7-62cd0a39f928" (UID: "6d9d7b07-acc4-44c1-9da7-62cd0a39f928"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:13:14.805202 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:14.805181    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83afef74-0242-4436-9d06-0dca98d8d5f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83afef74-0242-4436-9d06-0dca98d8d5f3" (UID: "83afef74-0242-4436-9d06-0dca98d8d5f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:13:14.906374 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:14.906345    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83afef74-0242-4436-9d06-0dca98d8d5f3-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:13:14.906374 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:14.906373    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d9d7b07-acc4-44c1-9da7-62cd0a39f928-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:13:15.525620 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:15.525590    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm"
Sep 09 04:13:15.525815 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:15.525633    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm" event={"ID":"6d9d7b07-acc4-44c1-9da7-62cd0a39f928","Type":"ContainerDied","Data":"593934b16c7f9b9558103592d694c16beff7d092682f02bab0e3a3f13865c3c6"}
Sep 09 04:13:15.525815 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:15.525717    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="593934b16c7f9b9558103592d694c16beff7d092682f02bab0e3a3f13865c3c6"
Sep 09 04:13:15.527061 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:15.527038    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv" event={"ID":"83afef74-0242-4436-9d06-0dca98d8d5f3","Type":"ContainerDied","Data":"6f7989b89c1843de6f076742737a26929960fdc942db5ec7033feee20d9ec647"}
Sep 09 04:13:15.527176 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:15.527069    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv"
Sep 09 04:13:15.527176 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:15.527078    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f7989b89c1843de6f076742737a26929960fdc942db5ec7033feee20d9ec647"
Sep 09 04:13:18.287467 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:18.287416    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:18.893563 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:18.893507    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:18.922622 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:18.922590    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"]
Sep 09 04:13:18.922854 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:18.922824    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct" podUID="7a6f5cc3-045c-4601-b99a-5493fc0774d1" containerName="payload-processing" containerID="cri-o://ecd679f382ebd968f77ddbce289bd04deb8cccd49912b7a6c97bf8230e499aa2" gracePeriod=30
Sep 09 04:13:18.933239 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:18.933213    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"]
Sep 09 04:13:18.934232 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:18.933462    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl" podUID="c0a53460-d7e4-4e29-9c31-fa7b4102a8a9" containerName="payload-pre-processing" containerID="cri-o://fbcaf548596656488d5a8aa16bee212b02c251b12ad845df6777a3162423dcc2" gracePeriod=30
Sep 09 04:13:18.967628 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:18.967599    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:19.026156 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.026132    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:19.094653 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.094596    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv"]
Sep 09 04:13:19.096866 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.096829    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:19.097295 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.097272    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1be20a5f-tcmlv"]
Sep 09 04:13:19.162482 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.162391    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:19.193853 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.193835    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"
Sep 09 04:13:19.218099 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.218072    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"
Sep 09 04:13:19.255476 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.255425    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:19.350943 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.350910    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6f5cc3-045c-4601-b99a-5493fc0774d1-plugins-config-volume\") pod \"7a6f5cc3-045c-4601-b99a-5493fc0774d1\" (UID: \"7a6f5cc3-045c-4601-b99a-5493fc0774d1\") "
Sep 09 04:13:19.351415 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.350959    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a53460-d7e4-4e29-9c31-fa7b4102a8a9-plugins-config-volume\") pod \"c0a53460-d7e4-4e29-9c31-fa7b4102a8a9\" (UID: \"c0a53460-d7e4-4e29-9c31-fa7b4102a8a9\") "
Sep 09 04:13:19.351415 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.351104    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg2qm\" (UniqueName: \"kubernetes.io/projected/c0a53460-d7e4-4e29-9c31-fa7b4102a8a9-kube-api-access-xg2qm\") pod \"c0a53460-d7e4-4e29-9c31-fa7b4102a8a9\" (UID: \"c0a53460-d7e4-4e29-9c31-fa7b4102a8a9\") "
Sep 09 04:13:19.351415 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.351136    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgkt9\" (UniqueName: \"kubernetes.io/projected/7a6f5cc3-045c-4601-b99a-5493fc0774d1-kube-api-access-bgkt9\") pod \"7a6f5cc3-045c-4601-b99a-5493fc0774d1\" (UID: \"7a6f5cc3-045c-4601-b99a-5493fc0774d1\") "
Sep 09 04:13:19.351415 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.351286    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6f5cc3-045c-4601-b99a-5493fc0774d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a6f5cc3-045c-4601-b99a-5493fc0774d1" (UID: "7a6f5cc3-045c-4601-b99a-5493fc0774d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:13:19.351415 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.351360    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a53460-d7e4-4e29-9c31-fa7b4102a8a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0a53460-d7e4-4e29-9c31-fa7b4102a8a9" (UID: "c0a53460-d7e4-4e29-9c31-fa7b4102a8a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:13:19.351849 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.351828    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6f5cc3-045c-4601-b99a-5493fc0774d1-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:13:19.352109 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.351852    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a53460-d7e4-4e29-9c31-fa7b4102a8a9-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:13:19.353706 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.353677    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6f5cc3-045c-4601-b99a-5493fc0774d1-kube-api-access-bgkt9" (OuterVolumeSpecName: "kube-api-access-bgkt9") pod "7a6f5cc3-045c-4601-b99a-5493fc0774d1" (UID: "7a6f5cc3-045c-4601-b99a-5493fc0774d1"). InnerVolumeSpecName "kube-api-access-bgkt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:13:19.353834 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.353814    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a53460-d7e4-4e29-9c31-fa7b4102a8a9-kube-api-access-xg2qm" (OuterVolumeSpecName: "kube-api-access-xg2qm") pod "c0a53460-d7e4-4e29-9c31-fa7b4102a8a9" (UID: "c0a53460-d7e4-4e29-9c31-fa7b4102a8a9"). InnerVolumeSpecName "kube-api-access-xg2qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:13:19.372783 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.372759    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:13:19.452433 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.452375    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg2qm\" (UniqueName: \"kubernetes.io/projected/c0a53460-d7e4-4e29-9c31-fa7b4102a8a9-kube-api-access-xg2qm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:13:19.452433 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.452398    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgkt9\" (UniqueName: \"kubernetes.io/projected/7a6f5cc3-045c-4601-b99a-5493fc0774d1-kube-api-access-bgkt9\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:13:19.551056 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.551030    2553 generic.go:356] "Generic (PLEG): container finished" podID="c0a53460-d7e4-4e29-9c31-fa7b4102a8a9" containerID="fbcaf548596656488d5a8aa16bee212b02c251b12ad845df6777a3162423dcc2" exitCode=0
Sep 09 04:13:19.551162 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.551074    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"
Sep 09 04:13:19.551232 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.551152    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl" event={"ID":"c0a53460-d7e4-4e29-9c31-fa7b4102a8a9","Type":"ContainerDied","Data":"fbcaf548596656488d5a8aa16bee212b02c251b12ad845df6777a3162423dcc2"}
Sep 09 04:13:19.551232 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.551204    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl" event={"ID":"c0a53460-d7e4-4e29-9c31-fa7b4102a8a9","Type":"ContainerDied","Data":"49f1fe7c4fd39ad1643806d2915c09fb3991cbc1aaa2253aab282ab4f28aa978"}
Sep 09 04:13:19.551337 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.551231    2553 scope.go:117] "RemoveContainer" containerID="fbcaf548596656488d5a8aa16bee212b02c251b12ad845df6777a3162423dcc2"
Sep 09 04:13:19.552519 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.552495    2553 generic.go:356] "Generic (PLEG): container finished" podID="7a6f5cc3-045c-4601-b99a-5493fc0774d1" containerID="ecd679f382ebd968f77ddbce289bd04deb8cccd49912b7a6c97bf8230e499aa2" exitCode=0
Sep 09 04:13:19.552610 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.552550    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"
Sep 09 04:13:19.552610 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.552565    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct" event={"ID":"7a6f5cc3-045c-4601-b99a-5493fc0774d1","Type":"ContainerDied","Data":"ecd679f382ebd968f77ddbce289bd04deb8cccd49912b7a6c97bf8230e499aa2"}
Sep 09 04:13:19.552610 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.552605    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct" event={"ID":"7a6f5cc3-045c-4601-b99a-5493fc0774d1","Type":"ContainerDied","Data":"a1f05e719e7ab23d6feb785af1c89ee837d07d87971bc1379683a7b083c2f25f"}
Sep 09 04:13:19.561669 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.561651    2553 scope.go:117] "RemoveContainer" containerID="fbcaf548596656488d5a8aa16bee212b02c251b12ad845df6777a3162423dcc2"
Sep 09 04:13:19.561942 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:13:19.561926    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbcaf548596656488d5a8aa16bee212b02c251b12ad845df6777a3162423dcc2\": container with ID starting with fbcaf548596656488d5a8aa16bee212b02c251b12ad845df6777a3162423dcc2 not found: ID does not exist" containerID="fbcaf548596656488d5a8aa16bee212b02c251b12ad845df6777a3162423dcc2"
Sep 09 04:13:19.562015 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.561953    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbcaf548596656488d5a8aa16bee212b02c251b12ad845df6777a3162423dcc2"} err="failed to get container status \"fbcaf548596656488d5a8aa16bee212b02c251b12ad845df6777a3162423dcc2\": rpc error: code = NotFound desc = could not find container \"fbcaf548596656488d5a8aa16bee212b02c251b12ad845df6777a3162423dcc2\": container with ID starting with fbcaf548596656488d5a8aa16bee212b02c251b12ad845df6777a3162423dcc2 not found: ID does not exist"
Sep 09 04:13:19.562015 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.561975    2553 scope.go:117] "RemoveContainer" containerID="ecd679f382ebd968f77ddbce289bd04deb8cccd49912b7a6c97bf8230e499aa2"
Sep 09 04:13:19.570315 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.570300    2553 scope.go:117] "RemoveContainer" containerID="ecd679f382ebd968f77ddbce289bd04deb8cccd49912b7a6c97bf8230e499aa2"
Sep 09 04:13:19.570603 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:13:19.570585    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecd679f382ebd968f77ddbce289bd04deb8cccd49912b7a6c97bf8230e499aa2\": container with ID starting with ecd679f382ebd968f77ddbce289bd04deb8cccd49912b7a6c97bf8230e499aa2 not found: ID does not exist" containerID="ecd679f382ebd968f77ddbce289bd04deb8cccd49912b7a6c97bf8230e499aa2"
Sep 09 04:13:19.570645 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.570609    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecd679f382ebd968f77ddbce289bd04deb8cccd49912b7a6c97bf8230e499aa2"} err="failed to get container status \"ecd679f382ebd968f77ddbce289bd04deb8cccd49912b7a6c97bf8230e499aa2\": rpc error: code = NotFound desc = could not find container \"ecd679f382ebd968f77ddbce289bd04deb8cccd49912b7a6c97bf8230e499aa2\": container with ID starting with ecd679f382ebd968f77ddbce289bd04deb8cccd49912b7a6c97bf8230e499aa2 not found: ID does not exist"
Sep 09 04:13:19.576724 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.576704    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"]
Sep 09 04:13:19.580743 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.580724    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1be20a5f-8577cc7b6f-nsxct"]
Sep 09 04:13:19.591107 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.591088    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"]
Sep 09 04:13:19.600886 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.598579    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1be20a5f-58649df44b-t6xxl"]
Sep 09 04:13:19.940593 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:19.940562    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:20.040946 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:20.040917    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:20.141244 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:20.141218    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:20.619390 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:20.619360    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:20.619794 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:20.619416    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:20.633128 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:20.633098    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:20.990270 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:20.990241    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a6f5cc3-045c-4601-b99a-5493fc0774d1" path="/var/lib/kubelet/pods/7a6f5cc3-045c-4601-b99a-5493fc0774d1/volumes"
Sep 09 04:13:20.990770 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:20.990747    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83afef74-0242-4436-9d06-0dca98d8d5f3" path="/var/lib/kubelet/pods/83afef74-0242-4436-9d06-0dca98d8d5f3/volumes"
Sep 09 04:13:20.991255 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:20.991236    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0a53460-d7e4-4e29-9c31-fa7b4102a8a9" path="/var/lib/kubelet/pods/c0a53460-d7e4-4e29-9c31-fa7b4102a8a9/volumes"
Sep 09 04:13:21.499864 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:21.499835    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:21.579977 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:21.579952    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:13:21.600773 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:21.600751    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:22.149858 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:22.149825    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:22.207994 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:22.207955    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:22.273288 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:22.273237    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:22.343569 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:22.343533    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm"]
Sep 09 04:13:22.345600 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:22.345576    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4ad5fa-kxvpm"]
Sep 09 04:13:22.346658 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:22.346635    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:22.408092 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:22.408026    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:22.694670 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:22.694594    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:22.989867 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:22.989842    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d9d7b07-acc4-44c1-9da7-62cd0a39f928" path="/var/lib/kubelet/pods/6d9d7b07-acc4-44c1-9da7-62cd0a39f928/volumes"
Sep 09 04:13:27.539670 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:27.539638    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:27.641600 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:27.641573    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:27.743167 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:27.743138    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:27.843377 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:27.843290    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:27.844228 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:27.844211    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"]
Sep 09 04:13:27.942268 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:27.942242    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:28.281545 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:28.281520    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:36.253832 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:36.253798    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:13:42.864582 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:42.864528    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn" podUID="23d766d4-c9a9-46ea-859e-dda2ac75d307" containerName="main" containerID="cri-o://f67d76ccb38c197f117a0992df3133616480f74998779cbf779e77526208ba62" gracePeriod=45
Sep 09 04:13:43.116778 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.116719    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:13:43.162892 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.162861    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-tmp-dir\") pod \"23d766d4-c9a9-46ea-859e-dda2ac75d307\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") "
Sep 09 04:13:43.163049 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.162936    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk2pr\" (UniqueName: \"kubernetes.io/projected/23d766d4-c9a9-46ea-859e-dda2ac75d307-kube-api-access-xk2pr\") pod \"23d766d4-c9a9-46ea-859e-dda2ac75d307\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") "
Sep 09 04:13:43.163049 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.162986    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-dshm\") pod \"23d766d4-c9a9-46ea-859e-dda2ac75d307\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") "
Sep 09 04:13:43.163049 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.163033    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23d766d4-c9a9-46ea-859e-dda2ac75d307-tls-certs\") pod \"23d766d4-c9a9-46ea-859e-dda2ac75d307\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") "
Sep 09 04:13:43.163229 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.163068    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-home\") pod \"23d766d4-c9a9-46ea-859e-dda2ac75d307\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") "
Sep 09 04:13:43.163229 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.163072    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23d766d4-c9a9-46ea-859e-dda2ac75d307" (UID: "23d766d4-c9a9-46ea-859e-dda2ac75d307"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:13:43.163229 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.163166    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-model-cache\") pod \"23d766d4-c9a9-46ea-859e-dda2ac75d307\" (UID: \"23d766d4-c9a9-46ea-859e-dda2ac75d307\") "
Sep 09 04:13:43.163383 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.163347    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-home" (OuterVolumeSpecName: "home") pod "23d766d4-c9a9-46ea-859e-dda2ac75d307" (UID: "23d766d4-c9a9-46ea-859e-dda2ac75d307"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:13:43.163595 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.163555    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-model-cache" (OuterVolumeSpecName: "model-cache") pod "23d766d4-c9a9-46ea-859e-dda2ac75d307" (UID: "23d766d4-c9a9-46ea-859e-dda2ac75d307"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:13:43.163855 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.163627    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:13:43.163855 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.163648    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:13:43.165391 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.165358    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-dshm" (OuterVolumeSpecName: "dshm") pod "23d766d4-c9a9-46ea-859e-dda2ac75d307" (UID: "23d766d4-c9a9-46ea-859e-dda2ac75d307"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:13:43.165508 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.165417    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d766d4-c9a9-46ea-859e-dda2ac75d307-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23d766d4-c9a9-46ea-859e-dda2ac75d307" (UID: "23d766d4-c9a9-46ea-859e-dda2ac75d307"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:13:43.165508 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.165432    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d766d4-c9a9-46ea-859e-dda2ac75d307-kube-api-access-xk2pr" (OuterVolumeSpecName: "kube-api-access-xk2pr") pod "23d766d4-c9a9-46ea-859e-dda2ac75d307" (UID: "23d766d4-c9a9-46ea-859e-dda2ac75d307"). InnerVolumeSpecName "kube-api-access-xk2pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:13:43.264404 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.264377    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23d766d4-c9a9-46ea-859e-dda2ac75d307-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:13:43.264404 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.264399    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:13:43.264404 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.264408    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk2pr\" (UniqueName: \"kubernetes.io/projected/23d766d4-c9a9-46ea-859e-dda2ac75d307-kube-api-access-xk2pr\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:13:43.264603 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.264415    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23d766d4-c9a9-46ea-859e-dda2ac75d307-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:13:43.684091 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.684059    2553 generic.go:356] "Generic (PLEG): container finished" podID="23d766d4-c9a9-46ea-859e-dda2ac75d307" containerID="f67d76ccb38c197f117a0992df3133616480f74998779cbf779e77526208ba62" exitCode=0
Sep 09 04:13:43.684242 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.684118    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"
Sep 09 04:13:43.684242 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.684126    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn" event={"ID":"23d766d4-c9a9-46ea-859e-dda2ac75d307","Type":"ContainerDied","Data":"f67d76ccb38c197f117a0992df3133616480f74998779cbf779e77526208ba62"}
Sep 09 04:13:43.684242 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.684162    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn" event={"ID":"23d766d4-c9a9-46ea-859e-dda2ac75d307","Type":"ContainerDied","Data":"543ee2a535172891823cef911c91b935b04e1e434ceb7901cab5fc3128b55e30"}
Sep 09 04:13:43.684242 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.684184    2553 scope.go:117] "RemoveContainer" containerID="f67d76ccb38c197f117a0992df3133616480f74998779cbf779e77526208ba62"
Sep 09 04:13:43.693312 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.693293    2553 scope.go:117] "RemoveContainer" containerID="f67d76ccb38c197f117a0992df3133616480f74998779cbf779e77526208ba62"
Sep 09 04:13:43.693594 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:13:43.693576    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67d76ccb38c197f117a0992df3133616480f74998779cbf779e77526208ba62\": container with ID starting with f67d76ccb38c197f117a0992df3133616480f74998779cbf779e77526208ba62 not found: ID does not exist" containerID="f67d76ccb38c197f117a0992df3133616480f74998779cbf779e77526208ba62"
Sep 09 04:13:43.693666 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.693606    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67d76ccb38c197f117a0992df3133616480f74998779cbf779e77526208ba62"} err="failed to get container status \"f67d76ccb38c197f117a0992df3133616480f74998779cbf779e77526208ba62\": rpc error: code = NotFound desc = could not find container \"f67d76ccb38c197f117a0992df3133616480f74998779cbf779e77526208ba62\": container with ID starting with f67d76ccb38c197f117a0992df3133616480f74998779cbf779e77526208ba62 not found: ID does not exist"
Sep 09 04:13:43.704764 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.704743    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"]
Sep 09 04:13:43.708688 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.708670    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-4ad5fa/autores-model-4ad5fa-kserve-6567875ccf-wv2kn"]
Sep 09 04:13:43.977295 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:43.977262    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:13:44.989438 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:44.989402    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23d766d4-c9a9-46ea-859e-dda2ac75d307" path="/var/lib/kubelet/pods/23d766d4-c9a9-46ea-859e-dda2ac75d307/volumes"
Sep 09 04:13:54.234655 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:13:54.234621    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:00.291014 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:00.290981    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:00.379173 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:00.379144    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:00.485032 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:00.485007    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:00.580794 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:00.580735    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:05.271758 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:05.271724    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:05.503198 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:05.503162    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"]
Sep 09 04:14:05.503491 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:05.503423    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq" podUID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" containerName="istio-proxy" containerID="cri-o://428f2056d4c1750d12152bdf15d04c1979d38c718002c93947e62207da303246" gracePeriod=30
Sep 09 04:14:05.517827 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:05.517804    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:05.961762 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:05.961725    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:06.250293 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:06.250268    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:06.419330 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:06.419300    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:08.982148 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:08.982114    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:09.082725 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:09.082694    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:10.127834 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.127801    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:10.228195 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.228160    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:10.329515 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.329488    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:10.697262 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.697200    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:10.757920 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.757900    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:14:10.789276 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.789202    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:10.796385 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.796363    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"]
Sep 09 04:14:10.796793 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.796773    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83afef74-0242-4436-9d06-0dca98d8d5f3" containerName="revoke-keys"
Sep 09 04:14:10.796793 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.796794    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="83afef74-0242-4436-9d06-0dca98d8d5f3" containerName="revoke-keys"
Sep 09 04:14:10.796942 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.796816    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d9d7b07-acc4-44c1-9da7-62cd0a39f928" containerName="revoke-keys"
Sep 09 04:14:10.796942 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.796826    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d9d7b07-acc4-44c1-9da7-62cd0a39f928" containerName="revoke-keys"
Sep 09 04:14:10.796942 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.796837    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0a53460-d7e4-4e29-9c31-fa7b4102a8a9" containerName="payload-pre-processing"
Sep 09 04:14:10.796942 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.796846    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0a53460-d7e4-4e29-9c31-fa7b4102a8a9" containerName="payload-pre-processing"
Sep 09 04:14:10.796942 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.796868    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d766d4-c9a9-46ea-859e-dda2ac75d307" containerName="main"
Sep 09 04:14:10.796942 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.796876    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d766d4-c9a9-46ea-859e-dda2ac75d307" containerName="main"
Sep 09 04:14:10.796942 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.796894    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" containerName="istio-proxy"
Sep 09 04:14:10.796942 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.796901    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" containerName="istio-proxy"
Sep 09 04:14:10.796942 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.796911    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6f5cc3-045c-4601-b99a-5493fc0774d1" containerName="payload-processing"
Sep 09 04:14:10.796942 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.796919    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6f5cc3-045c-4601-b99a-5493fc0774d1" containerName="payload-processing"
Sep 09 04:14:10.797252 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.797024    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="83afef74-0242-4436-9d06-0dca98d8d5f3" containerName="revoke-keys"
Sep 09 04:14:10.797252 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.797033    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6f5cc3-045c-4601-b99a-5493fc0774d1" containerName="payload-processing"
Sep 09 04:14:10.797252 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.797040    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d766d4-c9a9-46ea-859e-dda2ac75d307" containerName="main"
Sep 09 04:14:10.797252 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.797046    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" containerName="istio-proxy"
Sep 09 04:14:10.797252 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.797052    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d9d7b07-acc4-44c1-9da7-62cd0a39f928" containerName="revoke-keys"
Sep 09 04:14:10.797252 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.797059    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" containerName="istio-proxy"
Sep 09 04:14:10.797252 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.797067    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0a53460-d7e4-4e29-9c31-fa7b4102a8a9" containerName="payload-pre-processing"
Sep 09 04:14:10.800049 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.800033    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"
Sep 09 04:14:10.802310 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.802292    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b80654\""
Sep 09 04:14:10.802381 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.802350    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b80654-dockercfg-v2mdq\""
Sep 09 04:14:10.814096 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.814070    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"]
Sep 09 04:14:10.820860 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.820833    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"]
Sep 09 04:14:10.821762 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.821741    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" containerName="istio-proxy"
Sep 09 04:14:10.821762 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.821762    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" containerName="istio-proxy"
Sep 09 04:14:10.825000 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.824985    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"
Sep 09 04:14:10.829459 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.829406    2553 generic.go:356] "Generic (PLEG): container finished" podID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" containerID="428f2056d4c1750d12152bdf15d04c1979d38c718002c93947e62207da303246" exitCode=0
Sep 09 04:14:10.829657 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.829627    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq" event={"ID":"02557a9e-4916-4bdc-a1f2-b76f241ca2b0","Type":"ContainerDied","Data":"428f2056d4c1750d12152bdf15d04c1979d38c718002c93947e62207da303246"}
Sep 09 04:14:10.829896 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.829876    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq" event={"ID":"02557a9e-4916-4bdc-a1f2-b76f241ca2b0","Type":"ContainerDied","Data":"48066ab651e485ed543573c3f57ffd2d65b81b9a754a31e4ae420e6ad426e7c4"}
Sep 09 04:14:10.830230 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.829956    2553 scope.go:117] "RemoveContainer" containerID="428f2056d4c1750d12152bdf15d04c1979d38c718002c93947e62207da303246"
Sep 09 04:14:10.830466 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.830058    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"
Sep 09 04:14:10.833072 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.833044    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"]
Sep 09 04:14:10.842022 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.842001    2553 scope.go:117] "RemoveContainer" containerID="95d5a3d6bf7145498f45729ca9323c4bb5842c1e28dc64a5cdcb2bc3d0522a66"
Sep 09 04:14:10.851115 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.851100    2553 scope.go:117] "RemoveContainer" containerID="428f2056d4c1750d12152bdf15d04c1979d38c718002c93947e62207da303246"
Sep 09 04:14:10.851379 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:14:10.851363    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428f2056d4c1750d12152bdf15d04c1979d38c718002c93947e62207da303246\": container with ID starting with 428f2056d4c1750d12152bdf15d04c1979d38c718002c93947e62207da303246 not found: ID does not exist" containerID="428f2056d4c1750d12152bdf15d04c1979d38c718002c93947e62207da303246"
Sep 09 04:14:10.851426 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.851395    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428f2056d4c1750d12152bdf15d04c1979d38c718002c93947e62207da303246"} err="failed to get container status \"428f2056d4c1750d12152bdf15d04c1979d38c718002c93947e62207da303246\": rpc error: code = NotFound desc = could not find container \"428f2056d4c1750d12152bdf15d04c1979d38c718002c93947e62207da303246\": container with ID starting with 428f2056d4c1750d12152bdf15d04c1979d38c718002c93947e62207da303246 not found: ID does not exist"
Sep 09 04:14:10.851426 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.851414    2553 scope.go:117] "RemoveContainer" containerID="95d5a3d6bf7145498f45729ca9323c4bb5842c1e28dc64a5cdcb2bc3d0522a66"
Sep 09 04:14:10.851697 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:14:10.851680    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d5a3d6bf7145498f45729ca9323c4bb5842c1e28dc64a5cdcb2bc3d0522a66\": container with ID starting with 95d5a3d6bf7145498f45729ca9323c4bb5842c1e28dc64a5cdcb2bc3d0522a66 not found: ID does not exist" containerID="95d5a3d6bf7145498f45729ca9323c4bb5842c1e28dc64a5cdcb2bc3d0522a66"
Sep 09 04:14:10.851746 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.851703    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d5a3d6bf7145498f45729ca9323c4bb5842c1e28dc64a5cdcb2bc3d0522a66"} err="failed to get container status \"95d5a3d6bf7145498f45729ca9323c4bb5842c1e28dc64a5cdcb2bc3d0522a66\": rpc error: code = NotFound desc = could not find container \"95d5a3d6bf7145498f45729ca9323c4bb5842c1e28dc64a5cdcb2bc3d0522a66\": container with ID starting with 95d5a3d6bf7145498f45729ca9323c4bb5842c1e28dc64a5cdcb2bc3d0522a66 not found: ID does not exist"
Sep 09 04:14:10.907958 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.907933    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-token\") pod \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") "
Sep 09 04:14:10.908116 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.907975    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-credential-socket\") pod \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") "
Sep 09 04:14:10.908116 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.907998    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-workload-socket\") pod \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") "
Sep 09 04:14:10.908116 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.908024    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istiod-ca-cert\") pod \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") "
Sep 09 04:14:10.908316 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.908194    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-data\") pod \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") "
Sep 09 04:14:10.908316 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.908226    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-envoy\") pod \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") "
Sep 09 04:14:10.908316 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.908260    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-podinfo\") pod \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") "
Sep 09 04:14:10.908316 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.908293    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9b8s\" (UniqueName: \"kubernetes.io/projected/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-kube-api-access-s9b8s\") pod \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") "
Sep 09 04:14:10.910355 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.908318    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-workload-certs\") pod \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\" (UID: \"02557a9e-4916-4bdc-a1f2-b76f241ca2b0\") "
Sep 09 04:14:10.910355 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.908336    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "02557a9e-4916-4bdc-a1f2-b76f241ca2b0" (UID: "02557a9e-4916-4bdc-a1f2-b76f241ca2b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:14:10.910355 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.908585    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75fk6\" (UniqueName: \"kubernetes.io/projected/6adaa9f6-1047-429a-a5df-843391fbe386-kube-api-access-75fk6\") pod \"payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l\" (UID: \"6adaa9f6-1047-429a-a5df-843391fbe386\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"
Sep 09 04:14:10.910355 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.908797    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adaa9f6-1047-429a-a5df-843391fbe386-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l\" (UID: \"6adaa9f6-1047-429a-a5df-843391fbe386\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"
Sep 09 04:14:10.910355 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.908802    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "02557a9e-4916-4bdc-a1f2-b76f241ca2b0" (UID: "02557a9e-4916-4bdc-a1f2-b76f241ca2b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:14:10.910355 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.908841    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "02557a9e-4916-4bdc-a1f2-b76f241ca2b0" (UID: "02557a9e-4916-4bdc-a1f2-b76f241ca2b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:14:10.910355 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.908980    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-workload-socket\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:14:10.910355 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.909127    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "02557a9e-4916-4bdc-a1f2-b76f241ca2b0" (UID: "02557a9e-4916-4bdc-a1f2-b76f241ca2b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:14:10.910355 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.909252    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "02557a9e-4916-4bdc-a1f2-b76f241ca2b0" (UID: "02557a9e-4916-4bdc-a1f2-b76f241ca2b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:14:10.910355 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.909594    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istiod-ca-cert\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:14:10.910355 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.909618    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-data\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:14:10.912972 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.912282    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "02557a9e-4916-4bdc-a1f2-b76f241ca2b0" (UID: "02557a9e-4916-4bdc-a1f2-b76f241ca2b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:14:10.912972 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.912300    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "02557a9e-4916-4bdc-a1f2-b76f241ca2b0" (UID: "02557a9e-4916-4bdc-a1f2-b76f241ca2b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:14:10.913250 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.913211    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "02557a9e-4916-4bdc-a1f2-b76f241ca2b0" (UID: "02557a9e-4916-4bdc-a1f2-b76f241ca2b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:14:10.913812 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:10.913770    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-kube-api-access-s9b8s" (OuterVolumeSpecName: "kube-api-access-s9b8s") pod "02557a9e-4916-4bdc-a1f2-b76f241ca2b0" (UID: "02557a9e-4916-4bdc-a1f2-b76f241ca2b0"). InnerVolumeSpecName "kube-api-access-s9b8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:14:11.011019 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.010990    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adaa9f6-1047-429a-a5df-843391fbe386-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l\" (UID: \"6adaa9f6-1047-429a-a5df-843391fbe386\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"
Sep 09 04:14:11.011116 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.011049    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5847b049-7d25-447c-ac12-09fb4d131372-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx\" (UID: \"5847b049-7d25-447c-ac12-09fb4d131372\") " pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"
Sep 09 04:14:11.011116 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.011081    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrz9d\" (UniqueName: \"kubernetes.io/projected/5847b049-7d25-447c-ac12-09fb4d131372-kube-api-access-jrz9d\") pod \"payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx\" (UID: \"5847b049-7d25-447c-ac12-09fb4d131372\") " pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"
Sep 09 04:14:11.011116 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.011099    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75fk6\" (UniqueName: \"kubernetes.io/projected/6adaa9f6-1047-429a-a5df-843391fbe386-kube-api-access-75fk6\") pod \"payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l\" (UID: \"6adaa9f6-1047-429a-a5df-843391fbe386\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"
Sep 09 04:14:11.011315 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.011296    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-envoy\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:14:11.011364 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.011325    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-podinfo\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:14:11.011364 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.011339    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9b8s\" (UniqueName: \"kubernetes.io/projected/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-kube-api-access-s9b8s\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:14:11.011466 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.011356    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-workload-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:14:11.011466 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.011381    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-istio-token\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:14:11.011466 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.011393    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02557a9e-4916-4bdc-a1f2-b76f241ca2b0-credential-socket\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:14:11.011684 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.011669    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adaa9f6-1047-429a-a5df-843391fbe386-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l\" (UID: \"6adaa9f6-1047-429a-a5df-843391fbe386\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"
Sep 09 04:14:11.019801 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.019782    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75fk6\" (UniqueName: \"kubernetes.io/projected/6adaa9f6-1047-429a-a5df-843391fbe386-kube-api-access-75fk6\") pod \"payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l\" (UID: \"6adaa9f6-1047-429a-a5df-843391fbe386\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"
Sep 09 04:14:11.111636 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.111568    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"
Sep 09 04:14:11.111915 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.111897    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5847b049-7d25-447c-ac12-09fb4d131372-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx\" (UID: \"5847b049-7d25-447c-ac12-09fb4d131372\") " pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"
Sep 09 04:14:11.111973 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.111950    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrz9d\" (UniqueName: \"kubernetes.io/projected/5847b049-7d25-447c-ac12-09fb4d131372-kube-api-access-jrz9d\") pod \"payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx\" (UID: \"5847b049-7d25-447c-ac12-09fb4d131372\") " pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"
Sep 09 04:14:11.112616 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.112597    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5847b049-7d25-447c-ac12-09fb4d131372-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx\" (UID: \"5847b049-7d25-447c-ac12-09fb4d131372\") " pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"
Sep 09 04:14:11.119984 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.119966    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrz9d\" (UniqueName: \"kubernetes.io/projected/5847b049-7d25-447c-ac12-09fb4d131372-kube-api-access-jrz9d\") pod \"payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx\" (UID: \"5847b049-7d25-447c-ac12-09fb4d131372\") " pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"
Sep 09 04:14:11.137870 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.137843    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"
Sep 09 04:14:11.146095 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.146072    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"]
Sep 09 04:14:11.150131 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.150107    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-4ad5fa-openshift-default-86c6c8974d-56kkq"]
Sep 09 04:14:11.394508 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.394484    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"]
Sep 09 04:14:11.396212 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:14:11.396186    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6adaa9f6_1047_429a_a5df_843391fbe386.slice/crio-fda54c4d5dfbdf852d0ec3f17c5b95d8ce0970c5b9b1ca87accc99ff34d3d9c3 WatchSource:0}: Error finding container fda54c4d5dfbdf852d0ec3f17c5b95d8ce0970c5b9b1ca87accc99ff34d3d9c3: Status 404 returned error can't find the container with id fda54c4d5dfbdf852d0ec3f17c5b95d8ce0970c5b9b1ca87accc99ff34d3d9c3
Sep 09 04:14:11.428517 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.428497    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"]
Sep 09 04:14:11.431423 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:14:11.431399    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5847b049_7d25_447c_ac12_09fb4d131372.slice/crio-3ddbea1be2df6199919ec5e9bc20c24eedfed920f008d05bb874d6a9b2728f82 WatchSource:0}: Error finding container 3ddbea1be2df6199919ec5e9bc20c24eedfed920f008d05bb874d6a9b2728f82: Status 404 returned error can't find the container with id 3ddbea1be2df6199919ec5e9bc20c24eedfed920f008d05bb874d6a9b2728f82
Sep 09 04:14:11.836773 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.836740    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l" event={"ID":"6adaa9f6-1047-429a-a5df-843391fbe386","Type":"ContainerStarted","Data":"d6d37ffdfab2a63db397c41b6920d7d178e6d92e4a15c7effa8404f44a9527eb"}
Sep 09 04:14:11.836951 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.836801    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"
Sep 09 04:14:11.836951 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.836823    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l" event={"ID":"6adaa9f6-1047-429a-a5df-843391fbe386","Type":"ContainerStarted","Data":"fda54c4d5dfbdf852d0ec3f17c5b95d8ce0970c5b9b1ca87accc99ff34d3d9c3"}
Sep 09 04:14:11.838346 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.838323    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx" event={"ID":"5847b049-7d25-447c-ac12-09fb4d131372","Type":"ContainerStarted","Data":"f4e1e183a9e8e3e289118948676ea469821221a87d8f87acff626dd5be55c27d"}
Sep 09 04:14:11.838527 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.838360    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx" event={"ID":"5847b049-7d25-447c-ac12-09fb4d131372","Type":"ContainerStarted","Data":"3ddbea1be2df6199919ec5e9bc20c24eedfed920f008d05bb874d6a9b2728f82"}
Sep 09 04:14:11.838527 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.838432    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"
Sep 09 04:14:11.854202 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.854159    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l" podStartSLOduration=1.854145911 podStartE2EDuration="1.854145911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:14:11.852034977 +0000 UTC m=+951.378405687" watchObservedRunningTime="2026-09-09 04:14:11.854145911 +0000 UTC m=+951.380516621"
Sep 09 04:14:11.869165 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:11.869126    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx" podStartSLOduration=1.869112565 podStartE2EDuration="1.869112565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:14:11.86706278 +0000 UTC m=+951.393433491" watchObservedRunningTime="2026-09-09 04:14:11.869112565 +0000 UTC m=+951.395483273"
Sep 09 04:14:12.989076 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:12.989046    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02557a9e-4916-4bdc-a1f2-b76f241ca2b0" path="/var/lib/kubelet/pods/02557a9e-4916-4bdc-a1f2-b76f241ca2b0/volumes"
Sep 09 04:14:16.353875 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:16.353840    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:19.564950 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:19.564918    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:14:22.653568 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:22.653517    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:22.710712 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:22.710687    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:22.778769 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:22.778745    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:22.847950 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:22.847926    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"
Sep 09 04:14:22.848066 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:22.847990    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"
Sep 09 04:14:22.876750 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:22.876726    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:28.246335 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.246302    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"]
Sep 09 04:14:28.251594 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.251570    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.253997 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.253957    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-85732c\"/\"premium-w4-85732c-kserve-self-signed-certs\""
Sep 09 04:14:28.260599 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.260352    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"]
Sep 09 04:14:28.267079 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.267044    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:28.358297 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.358268    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197d60c8-9bf1-4af7-9739-857ae3d05438-tls-certs\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.358483 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.358306    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-model-cache\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.358483 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.358329    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-home\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.358483 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.358461    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgjb8\" (UniqueName: \"kubernetes.io/projected/197d60c8-9bf1-4af7-9739-857ae3d05438-kube-api-access-sgjb8\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.358620 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.358488    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-tmp-dir\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.358620 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.358524    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-dshm\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.459113 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.459082    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197d60c8-9bf1-4af7-9739-857ae3d05438-tls-certs\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.459305 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.459130    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-model-cache\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.459305 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.459163    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-home\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.459305 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.459222    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgjb8\" (UniqueName: \"kubernetes.io/projected/197d60c8-9bf1-4af7-9739-857ae3d05438-kube-api-access-sgjb8\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.459305 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.459243    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-tmp-dir\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.459305 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.459274    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-dshm\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.459681 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.459656    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-model-cache\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.459849 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.459695    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-home\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.460197 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.460161    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-tmp-dir\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.462094 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.462059    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197d60c8-9bf1-4af7-9739-857ae3d05438-tls-certs\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.462313 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.462290    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-dshm\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.467541 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.467523    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgjb8\" (UniqueName: \"kubernetes.io/projected/197d60c8-9bf1-4af7-9739-857ae3d05438-kube-api-access-sgjb8\") pod \"premium-w4-85732c-kserve-7f6fc7bb66-wfhbw\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") " pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.565891 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.565832    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:28.834645 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.834623    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"]
Sep 09 04:14:28.836385 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:14:28.836352    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod197d60c8_9bf1_4af7_9739_857ae3d05438.slice/crio-73cdb7048f0ef7461f9f2ccf4b86fb612332552e2516e3b4fd0c89151deb859c WatchSource:0}: Error finding container 73cdb7048f0ef7461f9f2ccf4b86fb612332552e2516e3b4fd0c89151deb859c: Status 404 returned error can't find the container with id 73cdb7048f0ef7461f9f2ccf4b86fb612332552e2516e3b4fd0c89151deb859c
Sep 09 04:14:28.937241 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.937211    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw" event={"ID":"197d60c8-9bf1-4af7-9739-857ae3d05438","Type":"ContainerStarted","Data":"540b9b2b20d8a56078a7ce7266171d785fa0b09b27dfb013fe6eb2cc5cc899da"}
Sep 09 04:14:28.937362 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.937258    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw" event={"ID":"197d60c8-9bf1-4af7-9739-857ae3d05438","Type":"ContainerStarted","Data":"73cdb7048f0ef7461f9f2ccf4b86fb612332552e2516e3b4fd0c89151deb859c"}
Sep 09 04:14:28.955565 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:28.955517    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw" podStartSLOduration=0.955501729 podStartE2EDuration="955.501729ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:14:28.954856652 +0000 UTC m=+968.481227353" watchObservedRunningTime="2026-09-09 04:14:28.955501729 +0000 UTC m=+968.481872437"
Sep 09 04:14:33.366900 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:33.366861    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:34.014074 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:34.014039    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:34.195269 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:34.195242    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:36.558308 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:36.558279    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:14:38.566195 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:38.566163    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:38.566585 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:38.566209    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:38.578574 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:38.578553    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:39.001866 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:39.001835    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:14:40.389861 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:40.389833    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:40.445171 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:40.445137    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:40.518332 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:40.518307    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:40.616292 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:40.616266    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:40.716347 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:40.716280    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:41.394467 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:41.394420    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:41.493228 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:41.493197    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:43.101890 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:43.101817    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:48.937798 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:48.937766    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:49.037345 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:49.037299    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:49.626744 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:49.626709    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:53.745726 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.745689    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"]
Sep 09 04:14:53.749290 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.749262    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.751365 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.751340    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-629522\"/\"kube-root-ca.crt\""
Sep 09 04:14:53.752385 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.752362    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-629522\"/\"openshift-service-ca.crt\""
Sep 09 04:14:53.752531 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.752504    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-629522\"/\"test-model-629522-kserve-self-signed-certs\""
Sep 09 04:14:53.752605 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.752594    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-629522\"/\"default-dockercfg-f42fn\""
Sep 09 04:14:53.759760 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.759743    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"]
Sep 09 04:14:53.779507 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.779486    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:53.887253 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.887226    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-home\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.887385 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.887263    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdkxv\" (UniqueName: \"kubernetes.io/projected/cc27e879-ae0c-41b5-872c-84453eb29fed-kube-api-access-pdkxv\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.887385 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.887290    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-dshm\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.887385 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.887372    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-tmp-dir\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.887535 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.887513    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc27e879-ae0c-41b5-872c-84453eb29fed-tls-certs\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.887572 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.887535    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-model-cache\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.988797 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.988771    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-home\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.988901 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.988804    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdkxv\" (UniqueName: \"kubernetes.io/projected/cc27e879-ae0c-41b5-872c-84453eb29fed-kube-api-access-pdkxv\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.988901 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.988827    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-dshm\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.988901 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.988851    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-tmp-dir\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.989017 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.988949    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc27e879-ae0c-41b5-872c-84453eb29fed-tls-certs\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.989017 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.988975    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-model-cache\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.989202 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.989182    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-home\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.989299 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.989272    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-tmp-dir\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.989374 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.989304    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-model-cache\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.991279 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.991257    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-dshm\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.991400 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.991384    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc27e879-ae0c-41b5-872c-84453eb29fed-tls-certs\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:53.996328 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:53.996275    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdkxv\" (UniqueName: \"kubernetes.io/projected/cc27e879-ae0c-41b5-872c-84453eb29fed-kube-api-access-pdkxv\") pod \"test-model-629522-kserve-7879b99c97-52vvp\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") " pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:54.060572 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:54.060551    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:14:54.334037 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:54.334011    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"]
Sep 09 04:14:54.335903 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:14:54.335872    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc27e879_ae0c_41b5_872c_84453eb29fed.slice/crio-6c1b86d92dbad471c27a42aec6b211fbc8f8c2bc857d79d2bb9a9bcf3ccf48dc WatchSource:0}: Error finding container 6c1b86d92dbad471c27a42aec6b211fbc8f8c2bc857d79d2bb9a9bcf3ccf48dc: Status 404 returned error can't find the container with id 6c1b86d92dbad471c27a42aec6b211fbc8f8c2bc857d79d2bb9a9bcf3ccf48dc
Sep 09 04:14:55.071998 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:55.071963    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp" event={"ID":"cc27e879-ae0c-41b5-872c-84453eb29fed","Type":"ContainerStarted","Data":"78b3fb871dfb5a93312a869ccab1bba079167a6801a414e5e64e958ce2b5babe"}
Sep 09 04:14:55.072353 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:55.072013    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp" event={"ID":"cc27e879-ae0c-41b5-872c-84453eb29fed","Type":"ContainerStarted","Data":"6c1b86d92dbad471c27a42aec6b211fbc8f8c2bc857d79d2bb9a9bcf3ccf48dc"}
Sep 09 04:14:55.090418 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:55.090363    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp" podStartSLOduration=2.0903427900000002 podStartE2EDuration="2.09034279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:14:55.087794867 +0000 UTC m=+994.614165576" watchObservedRunningTime="2026-09-09 04:14:55.09034279 +0000 UTC m=+994.616713495"
Sep 09 04:14:55.850622 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:55.850589    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:55.948351 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:55.948316    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:56.094655 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:56.094621    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:56.381975 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:56.381945    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:56.754321 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:56.754285    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:58.557367 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:58.557316    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:14:59.291192 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:59.291161    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:14:59.384857 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:14:59.384830    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:00.143410 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.143381    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b80654-29815455-ffd2x"]
Sep 09 04:15:00.147114 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.147097    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815455-7shl5"]
Sep 09 04:15:00.147225 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.147208    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b80654-29815455-ffd2x"
Sep 09 04:15:00.149682 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.149661    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:15:00.150148 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.150124    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815455-z72hj"]
Sep 09 04:15:00.150285 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.150264    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815455-7shl5"
Sep 09 04:15:00.154098 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.154068    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b80654-29815455-ffd2x"]
Sep 09 04:15:00.154211 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.154168    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815455-z72hj"
Sep 09 04:15:00.157251 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.157226    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815455-7shl5"]
Sep 09 04:15:00.165252 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.165232    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b80654-29815455-ffd2x"
Sep 09 04:15:00.174223 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.173911    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815455-z72hj"]
Sep 09 04:15:00.178970 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.178395    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815455-7shl5"
Sep 09 04:15:00.226878 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.226850    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815455-z72hj"
Sep 09 04:15:00.465390 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.465234    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b80654-29815455-ffd2x"]
Sep 09 04:15:00.471866 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.471840    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815455-7shl5"]
Sep 09 04:15:00.473869 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:15:00.473844    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6913ea88_2577_4ad2_844e_cc5ce75097b2.slice/crio-2a46df990d435136a4facf785b164f145fca2a869bdb3a66cb49287673420800 WatchSource:0}: Error finding container 2a46df990d435136a4facf785b164f145fca2a869bdb3a66cb49287673420800: Status 404 returned error can't find the container with id 2a46df990d435136a4facf785b164f145fca2a869bdb3a66cb49287673420800
Sep 09 04:15:00.517293 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.517260    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:00.531190 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.531168    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815455-z72hj"]
Sep 09 04:15:00.533270 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:15:00.533245    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f44517_07ec_4d4c_8a7f_fbff9f306ccf.slice/crio-53a9e88f922ab62085a762adb0a218a5ed61a1162590e4463004440724c4e9cc WatchSource:0}: Error finding container 53a9e88f922ab62085a762adb0a218a5ed61a1162590e4463004440724c4e9cc: Status 404 returned error can't find the container with id 53a9e88f922ab62085a762adb0a218a5ed61a1162590e4463004440724c4e9cc
Sep 09 04:15:00.616963 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.616937    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:00.775081 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.775053    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:00.992657 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:00.992627    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:01.094953 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.094870    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:01.105384 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.105350    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815455-7shl5" event={"ID":"6913ea88-2577-4ad2-844e-cc5ce75097b2","Type":"ContainerStarted","Data":"3abda57d54a46a7896ee22e22fb636fdd5c56ac320df6afcd11f5367f2bec74d"}
Sep 09 04:15:01.105557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.105413    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815455-7shl5" event={"ID":"6913ea88-2577-4ad2-844e-cc5ce75097b2","Type":"ContainerStarted","Data":"2a46df990d435136a4facf785b164f145fca2a869bdb3a66cb49287673420800"}
Sep 09 04:15:01.107288 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.107263    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b80654-29815455-ffd2x" event={"ID":"669c4bab-d7df-4ff7-9a19-26f7ed3093de","Type":"ContainerStarted","Data":"c1bde97f2ca8720ff6f2044ac67b0ca43b53ffff79ccaa777cdf4957581efcb6"}
Sep 09 04:15:01.107416 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.107315    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b80654-29815455-ffd2x" event={"ID":"669c4bab-d7df-4ff7-9a19-26f7ed3093de","Type":"ContainerStarted","Data":"b688b13ad77760c447a155f8156b0eefe778d2c23fa06f604265eccadb58957a"}
Sep 09 04:15:01.108964 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.108940    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815455-z72hj" event={"ID":"e4f44517-07ec-4d4c-8a7f-fbff9f306ccf","Type":"ContainerStarted","Data":"9c59f4ff7f054a692a2d7b487cdcfee391215788b7a07e68130f6d63c5e2115d"}
Sep 09 04:15:01.109077 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.108985    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815455-z72hj" event={"ID":"e4f44517-07ec-4d4c-8a7f-fbff9f306ccf","Type":"ContainerStarted","Data":"53a9e88f922ab62085a762adb0a218a5ed61a1162590e4463004440724c4e9cc"}
Sep 09 04:15:01.124609 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.124570    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"]
Sep 09 04:15:01.128897 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.128861    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815455-7shl5" podStartSLOduration=1.128848276 podStartE2EDuration="1.128848276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:15:01.126571969 +0000 UTC m=+1000.652942678" watchObservedRunningTime="2026-09-09 04:15:01.128848276 +0000 UTC m=+1000.655218985"
Sep 09 04:15:01.129623 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.129602    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"
Sep 09 04:15:01.132080 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.132052    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f0f5e1c0-dockercfg-tdgnr\""
Sep 09 04:15:01.132194 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.132060    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f0f5e1c0\""
Sep 09 04:15:01.135667 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.135648    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"]
Sep 09 04:15:01.144653 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.144610    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815455-z72hj" podStartSLOduration=1.144592749 podStartE2EDuration="1.144592749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:15:01.143736398 +0000 UTC m=+1000.670107109" watchObservedRunningTime="2026-09-09 04:15:01.144592749 +0000 UTC m=+1000.670963458"
Sep 09 04:15:01.161192 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.161166    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/597197a3-059f-46d2-957b-40c688ae95ff-plugins-config-volume\") pod \"payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq\" (UID: \"597197a3-059f-46d2-957b-40c688ae95ff\") " pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"
Sep 09 04:15:01.161303 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.161250    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b80654-29815455-ffd2x" podStartSLOduration=1.161237882 podStartE2EDuration="1.161237882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:15:01.160193695 +0000 UTC m=+1000.686564400" watchObservedRunningTime="2026-09-09 04:15:01.161237882 +0000 UTC m=+1000.687608591"
Sep 09 04:15:01.161359 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.161337    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnzpq\" (UniqueName: \"kubernetes.io/projected/597197a3-059f-46d2-957b-40c688ae95ff-kube-api-access-cnzpq\") pod \"payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq\" (UID: \"597197a3-059f-46d2-957b-40c688ae95ff\") " pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"
Sep 09 04:15:01.262848 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.262811    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzpq\" (UniqueName: \"kubernetes.io/projected/597197a3-059f-46d2-957b-40c688ae95ff-kube-api-access-cnzpq\") pod \"payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq\" (UID: \"597197a3-059f-46d2-957b-40c688ae95ff\") " pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"
Sep 09 04:15:01.262989 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.262899    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/597197a3-059f-46d2-957b-40c688ae95ff-plugins-config-volume\") pod \"payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq\" (UID: \"597197a3-059f-46d2-957b-40c688ae95ff\") " pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"
Sep 09 04:15:01.263595 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.263578    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/597197a3-059f-46d2-957b-40c688ae95ff-plugins-config-volume\") pod \"payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq\" (UID: \"597197a3-059f-46d2-957b-40c688ae95ff\") " pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"
Sep 09 04:15:01.270745 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.270724    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzpq\" (UniqueName: \"kubernetes.io/projected/597197a3-059f-46d2-957b-40c688ae95ff-kube-api-access-cnzpq\") pod \"payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq\" (UID: \"597197a3-059f-46d2-957b-40c688ae95ff\") " pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"
Sep 09 04:15:01.442189 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.442105    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"
Sep 09 04:15:01.712474 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:01.712431    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"]
Sep 09 04:15:01.788158 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:15:01.788131    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597197a3_059f_46d2_957b_40c688ae95ff.slice/crio-9aa2c4eab4edecaa01a58c54910b8269bb8210cf57ccdad6e472ac53bf43caec WatchSource:0}: Error finding container 9aa2c4eab4edecaa01a58c54910b8269bb8210cf57ccdad6e472ac53bf43caec: Status 404 returned error can't find the container with id 9aa2c4eab4edecaa01a58c54910b8269bb8210cf57ccdad6e472ac53bf43caec
Sep 09 04:15:02.117171 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:02.117137    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq" event={"ID":"597197a3-059f-46d2-957b-40c688ae95ff","Type":"ContainerStarted","Data":"a387fcbb59d24bc9efcabbfb8168ef8be212d3dc0243a7dc7b5011d6a91cf96f"}
Sep 09 04:15:02.117373 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:02.117198    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"
Sep 09 04:15:02.117373 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:02.117223    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq" event={"ID":"597197a3-059f-46d2-957b-40c688ae95ff","Type":"ContainerStarted","Data":"9aa2c4eab4edecaa01a58c54910b8269bb8210cf57ccdad6e472ac53bf43caec"}
Sep 09 04:15:02.118797 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:02.118776    2553 generic.go:356] "Generic (PLEG): container finished" podID="e4f44517-07ec-4d4c-8a7f-fbff9f306ccf" containerID="9c59f4ff7f054a692a2d7b487cdcfee391215788b7a07e68130f6d63c5e2115d" exitCode=0
Sep 09 04:15:02.118905 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:02.118850    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815455-z72hj" event={"ID":"e4f44517-07ec-4d4c-8a7f-fbff9f306ccf","Type":"ContainerDied","Data":"9c59f4ff7f054a692a2d7b487cdcfee391215788b7a07e68130f6d63c5e2115d"}
Sep 09 04:15:02.120312 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:02.120288    2553 generic.go:356] "Generic (PLEG): container finished" podID="6913ea88-2577-4ad2-844e-cc5ce75097b2" containerID="3abda57d54a46a7896ee22e22fb636fdd5c56ac320df6afcd11f5367f2bec74d" exitCode=0
Sep 09 04:15:02.120406 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:02.120372    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815455-7shl5" event={"ID":"6913ea88-2577-4ad2-844e-cc5ce75097b2","Type":"ContainerDied","Data":"3abda57d54a46a7896ee22e22fb636fdd5c56ac320df6afcd11f5367f2bec74d"}
Sep 09 04:15:02.121833 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:02.121811    2553 generic.go:356] "Generic (PLEG): container finished" podID="669c4bab-d7df-4ff7-9a19-26f7ed3093de" containerID="c1bde97f2ca8720ff6f2044ac67b0ca43b53ffff79ccaa777cdf4957581efcb6" exitCode=0
Sep 09 04:15:02.121938 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:02.121890    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b80654-29815455-ffd2x" event={"ID":"669c4bab-d7df-4ff7-9a19-26f7ed3093de","Type":"ContainerDied","Data":"c1bde97f2ca8720ff6f2044ac67b0ca43b53ffff79ccaa777cdf4957581efcb6"}
Sep 09 04:15:02.133760 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:02.133712    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq" podStartSLOduration=1.1336947880000001 podStartE2EDuration="1.133694788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:15:02.133601792 +0000 UTC m=+1001.659972520" watchObservedRunningTime="2026-09-09 04:15:02.133694788 +0000 UTC m=+1001.660065501"
Sep 09 04:15:03.279123 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:03.278983    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815455-z72hj"
Sep 09 04:15:03.333175 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:03.333153    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815455-7shl5"
Sep 09 04:15:03.337242 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:03.337224    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b80654-29815455-ffd2x"
Sep 09 04:15:04.061377 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:04.061350    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:15:04.061377 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:04.061388    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:15:04.073576 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:04.073550    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:15:04.138323 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:04.138298    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815455-z72hj"
Sep 09 04:15:04.138323 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:04.138315    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815455-z72hj" event={"ID":"e4f44517-07ec-4d4c-8a7f-fbff9f306ccf","Type":"ContainerDied","Data":"53a9e88f922ab62085a762adb0a218a5ed61a1162590e4463004440724c4e9cc"}
Sep 09 04:15:04.138542 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:04.138356    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53a9e88f922ab62085a762adb0a218a5ed61a1162590e4463004440724c4e9cc"
Sep 09 04:15:04.139695 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:04.139677    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815455-7shl5"
Sep 09 04:15:04.139695 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:04.139683    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815455-7shl5" event={"ID":"6913ea88-2577-4ad2-844e-cc5ce75097b2","Type":"ContainerDied","Data":"2a46df990d435136a4facf785b164f145fca2a869bdb3a66cb49287673420800"}
Sep 09 04:15:04.139848 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:04.139721    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a46df990d435136a4facf785b164f145fca2a869bdb3a66cb49287673420800"
Sep 09 04:15:04.141101 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:04.141085    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b80654-29815455-ffd2x"
Sep 09 04:15:04.141193 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:04.141100    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b80654-29815455-ffd2x" event={"ID":"669c4bab-d7df-4ff7-9a19-26f7ed3093de","Type":"ContainerDied","Data":"b688b13ad77760c447a155f8156b0eefe778d2c23fa06f604265eccadb58957a"}
Sep 09 04:15:04.141193 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:04.141139    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b688b13ad77760c447a155f8156b0eefe778d2c23fa06f604265eccadb58957a"
Sep 09 04:15:04.153155 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:04.153128    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:15:13.129602 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:13.129575    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"
Sep 09 04:15:23.060853 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:23.060826    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:15:24.381095 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:24.381056    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:24.917602 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:24.917563    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:25.016601 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:25.016570    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:25.368285 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:25.368261    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:31.438260 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.438224    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:31.507263 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.507226    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:31.786591 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.786563    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp"]
Sep 09 04:15:31.786992 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.786980    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="669c4bab-d7df-4ff7-9a19-26f7ed3093de" containerName="cleanup"
Sep 09 04:15:31.787037 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.786994    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="669c4bab-d7df-4ff7-9a19-26f7ed3093de" containerName="cleanup"
Sep 09 04:15:31.787037 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.787016    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f44517-07ec-4d4c-8a7f-fbff9f306ccf" containerName="cleanup"
Sep 09 04:15:31.787037 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.787021    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f44517-07ec-4d4c-8a7f-fbff9f306ccf" containerName="cleanup"
Sep 09 04:15:31.787037 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.787029    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6913ea88-2577-4ad2-844e-cc5ce75097b2" containerName="cleanup"
Sep 09 04:15:31.787037 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.787034    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6913ea88-2577-4ad2-844e-cc5ce75097b2" containerName="cleanup"
Sep 09 04:15:31.787193 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.787101    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4f44517-07ec-4d4c-8a7f-fbff9f306ccf" containerName="cleanup"
Sep 09 04:15:31.787193 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.787108    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6913ea88-2577-4ad2-844e-cc5ce75097b2" containerName="cleanup"
Sep 09 04:15:31.787193 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.787114    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="669c4bab-d7df-4ff7-9a19-26f7ed3093de" containerName="cleanup"
Sep 09 04:15:31.791370 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.791354    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp"
Sep 09 04:15:31.793685 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.793657    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:15:31.793798 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.793705    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:15:31.799072 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.799052    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp"]
Sep 09 04:15:31.932169 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:31.932138    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5dd30b-93bd-4fc6-9f6b-daeee16db106-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp\" (UID: \"be5dd30b-93bd-4fc6-9f6b-daeee16db106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp"
Sep 09 04:15:32.033581 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:32.033553    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5dd30b-93bd-4fc6-9f6b-daeee16db106-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp\" (UID: \"be5dd30b-93bd-4fc6-9f6b-daeee16db106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp"
Sep 09 04:15:32.034153 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:32.034133    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5dd30b-93bd-4fc6-9f6b-daeee16db106-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp\" (UID: \"be5dd30b-93bd-4fc6-9f6b-daeee16db106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp"
Sep 09 04:15:32.103027 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:32.102953    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp"
Sep 09 04:15:32.378520 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:32.378382    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp"]
Sep 09 04:15:32.386189 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:15:32.386159    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5dd30b_93bd_4fc6_9f6b_daeee16db106.slice/crio-8ba5ced240356f0318cc72809f0046ab9e1692b5a301d5cd84cefd233d88ec6e WatchSource:0}: Error finding container 8ba5ced240356f0318cc72809f0046ab9e1692b5a301d5cd84cefd233d88ec6e: Status 404 returned error can't find the container with id 8ba5ced240356f0318cc72809f0046ab9e1692b5a301d5cd84cefd233d88ec6e
Sep 09 04:15:33.100283 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:15:33.100254    2553 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5dd30b_93bd_4fc6_9f6b_daeee16db106.slice/crio-5348aed62fc370d21dda8cbe5b1a29e79e9dc94adba44a3ecd17ec5692706672.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 04:15:33.299523 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:33.299495    2553 generic.go:356] "Generic (PLEG): container finished" podID="be5dd30b-93bd-4fc6-9f6b-daeee16db106" containerID="5348aed62fc370d21dda8cbe5b1a29e79e9dc94adba44a3ecd17ec5692706672" exitCode=0
Sep 09 04:15:33.299668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:33.299580    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp" event={"ID":"be5dd30b-93bd-4fc6-9f6b-daeee16db106","Type":"ContainerDied","Data":"5348aed62fc370d21dda8cbe5b1a29e79e9dc94adba44a3ecd17ec5692706672"}
Sep 09 04:15:33.299668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:33.299625    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp" event={"ID":"be5dd30b-93bd-4fc6-9f6b-daeee16db106","Type":"ContainerStarted","Data":"8ba5ced240356f0318cc72809f0046ab9e1692b5a301d5cd84cefd233d88ec6e"}
Sep 09 04:15:34.438188 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.438164    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp"
Sep 09 04:15:34.462638 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.462608    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"]
Sep 09 04:15:34.463289 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.463271    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be5dd30b-93bd-4fc6-9f6b-daeee16db106" containerName="revoke-keys"
Sep 09 04:15:34.463360 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.463291    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="be5dd30b-93bd-4fc6-9f6b-daeee16db106" containerName="revoke-keys"
Sep 09 04:15:34.463360 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.463356    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="be5dd30b-93bd-4fc6-9f6b-daeee16db106" containerName="revoke-keys"
Sep 09 04:15:34.466390 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.466373    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.468893 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.468872    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b80654\"/\"default-dockercfg-dltbj\""
Sep 09 04:15:34.469049 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.469028    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b80654\"/\"kube-root-ca.crt\""
Sep 09 04:15:34.469174 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.469029    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b80654\"/\"openshift-service-ca.crt\""
Sep 09 04:15:34.469174 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.468984    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b80654\"/\"explicit-model-b80654-kserve-self-signed-certs\""
Sep 09 04:15:34.475952 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.475803    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"]
Sep 09 04:15:34.491403 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.491382    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:34.554704 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.554680    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5dd30b-93bd-4fc6-9f6b-daeee16db106-maas-api-service-ca\") pod \"be5dd30b-93bd-4fc6-9f6b-daeee16db106\" (UID: \"be5dd30b-93bd-4fc6-9f6b-daeee16db106\") "
Sep 09 04:15:34.555027 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.555007    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-dshm\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.555069 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.555022    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be5dd30b-93bd-4fc6-9f6b-daeee16db106-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be5dd30b-93bd-4fc6-9f6b-daeee16db106" (UID: "be5dd30b-93bd-4fc6-9f6b-daeee16db106"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:15:34.555069 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.555049    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-tls-certs\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.555142 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.555077    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-model-cache\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.555142 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.555105    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76pfk\" (UniqueName: \"kubernetes.io/projected/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-kube-api-access-76pfk\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.555218 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.555201    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-tmp-dir\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.555333 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.555296    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-home\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.555371 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.555348    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5dd30b-93bd-4fc6-9f6b-daeee16db106-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:15:34.656542 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.656511    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76pfk\" (UniqueName: \"kubernetes.io/projected/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-kube-api-access-76pfk\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.656699 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.656576    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-tmp-dir\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.656699 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.656686    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-home\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.656825 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.656749    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-dshm\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.656825 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.656781    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-tls-certs\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.656825 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.656815    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-model-cache\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.657023 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.657002    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-tmp-dir\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.657096 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.657076    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-home\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.657157 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.657118    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-model-cache\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.659256 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.659235    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-dshm\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.659520 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.659505    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-tls-certs\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.664695 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.664678    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76pfk\" (UniqueName: \"kubernetes.io/projected/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-kube-api-access-76pfk\") pod \"explicit-model-b80654-kserve-7c78576c79-94pzw\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") " pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:34.779502 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:34.779483    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:35.050533 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:35.050505    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"]
Sep 09 04:15:35.051643 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:15:35.051618    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65bbd4fc_ec6d_4c5d_ae91_c01f8acc32ba.slice/crio-691880e63198c24c1b4d67afa118bd01f251424b1378d60829295031f668d02c WatchSource:0}: Error finding container 691880e63198c24c1b4d67afa118bd01f251424b1378d60829295031f668d02c: Status 404 returned error can't find the container with id 691880e63198c24c1b4d67afa118bd01f251424b1378d60829295031f668d02c
Sep 09 04:15:35.312239 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:35.312213    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp"
Sep 09 04:15:35.312413 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:35.312249    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp" event={"ID":"be5dd30b-93bd-4fc6-9f6b-daeee16db106","Type":"ContainerDied","Data":"8ba5ced240356f0318cc72809f0046ab9e1692b5a301d5cd84cefd233d88ec6e"}
Sep 09 04:15:35.312413 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:35.312291    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ba5ced240356f0318cc72809f0046ab9e1692b5a301d5cd84cefd233d88ec6e"
Sep 09 04:15:35.314044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:35.314009    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw" event={"ID":"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba","Type":"ContainerStarted","Data":"5a89e0a1425ab74eefad2d3d19122d8afb69906e458b898ab1730f028d596969"}
Sep 09 04:15:35.314044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:35.314046    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw" event={"ID":"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba","Type":"ContainerStarted","Data":"691880e63198c24c1b4d67afa118bd01f251424b1378d60829295031f668d02c"}
Sep 09 04:15:35.331676 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:35.331605    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw" podStartSLOduration=1.331591908 podStartE2EDuration="1.331591908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:15:35.330163082 +0000 UTC m=+1034.856533793" watchObservedRunningTime="2026-09-09 04:15:35.331591908 +0000 UTC m=+1034.857962619"
Sep 09 04:15:42.271733 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.271695    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:42.303725 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.303694    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"]
Sep 09 04:15:42.303961 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.303938    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq" podUID="597197a3-059f-46d2-957b-40c688ae95ff" containerName="payload-processing" containerID="cri-o://a387fcbb59d24bc9efcabbfb8168ef8be212d3dc0243a7dc7b5011d6a91cf96f" gracePeriod=30
Sep 09 04:15:42.370894 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.370861    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:42.473149 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.473115    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:42.473820 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.473797    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp"]
Sep 09 04:15:42.476320 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.476297    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0f5e1c0-sp4mp"]
Sep 09 04:15:42.539109 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.539032    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:42.584846 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.584822    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"
Sep 09 04:15:42.634162 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.634135    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnzpq\" (UniqueName: \"kubernetes.io/projected/597197a3-059f-46d2-957b-40c688ae95ff-kube-api-access-cnzpq\") pod \"597197a3-059f-46d2-957b-40c688ae95ff\" (UID: \"597197a3-059f-46d2-957b-40c688ae95ff\") "
Sep 09 04:15:42.634318 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.634244    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/597197a3-059f-46d2-957b-40c688ae95ff-plugins-config-volume\") pod \"597197a3-059f-46d2-957b-40c688ae95ff\" (UID: \"597197a3-059f-46d2-957b-40c688ae95ff\") "
Sep 09 04:15:42.634668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.634649    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/597197a3-059f-46d2-957b-40c688ae95ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "597197a3-059f-46d2-957b-40c688ae95ff" (UID: "597197a3-059f-46d2-957b-40c688ae95ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:15:42.636304 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.636283    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597197a3-059f-46d2-957b-40c688ae95ff-kube-api-access-cnzpq" (OuterVolumeSpecName: "kube-api-access-cnzpq") pod "597197a3-059f-46d2-957b-40c688ae95ff" (UID: "597197a3-059f-46d2-957b-40c688ae95ff"). InnerVolumeSpecName "kube-api-access-cnzpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:15:42.736051 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.736023    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/597197a3-059f-46d2-957b-40c688ae95ff-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:15:42.736051 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.736052    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnzpq\" (UniqueName: \"kubernetes.io/projected/597197a3-059f-46d2-957b-40c688ae95ff-kube-api-access-cnzpq\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:15:42.830292 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.830227    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:42.991011 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:42.990978    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be5dd30b-93bd-4fc6-9f6b-daeee16db106" path="/var/lib/kubelet/pods/be5dd30b-93bd-4fc6-9f6b-daeee16db106/volumes"
Sep 09 04:15:43.364924 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:43.364892    2553 generic.go:356] "Generic (PLEG): container finished" podID="597197a3-059f-46d2-957b-40c688ae95ff" containerID="a387fcbb59d24bc9efcabbfb8168ef8be212d3dc0243a7dc7b5011d6a91cf96f" exitCode=0
Sep 09 04:15:43.365339 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:43.364934    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"
Sep 09 04:15:43.365339 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:43.364974    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq" event={"ID":"597197a3-059f-46d2-957b-40c688ae95ff","Type":"ContainerDied","Data":"a387fcbb59d24bc9efcabbfb8168ef8be212d3dc0243a7dc7b5011d6a91cf96f"}
Sep 09 04:15:43.365339 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:43.365028    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq" event={"ID":"597197a3-059f-46d2-957b-40c688ae95ff","Type":"ContainerDied","Data":"9aa2c4eab4edecaa01a58c54910b8269bb8210cf57ccdad6e472ac53bf43caec"}
Sep 09 04:15:43.365339 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:43.365055    2553 scope.go:117] "RemoveContainer" containerID="a387fcbb59d24bc9efcabbfb8168ef8be212d3dc0243a7dc7b5011d6a91cf96f"
Sep 09 04:15:43.374010 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:43.373995    2553 scope.go:117] "RemoveContainer" containerID="a387fcbb59d24bc9efcabbfb8168ef8be212d3dc0243a7dc7b5011d6a91cf96f"
Sep 09 04:15:43.374281 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:15:43.374263    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a387fcbb59d24bc9efcabbfb8168ef8be212d3dc0243a7dc7b5011d6a91cf96f\": container with ID starting with a387fcbb59d24bc9efcabbfb8168ef8be212d3dc0243a7dc7b5011d6a91cf96f not found: ID does not exist" containerID="a387fcbb59d24bc9efcabbfb8168ef8be212d3dc0243a7dc7b5011d6a91cf96f"
Sep 09 04:15:43.374320 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:43.374290    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a387fcbb59d24bc9efcabbfb8168ef8be212d3dc0243a7dc7b5011d6a91cf96f"} err="failed to get container status \"a387fcbb59d24bc9efcabbfb8168ef8be212d3dc0243a7dc7b5011d6a91cf96f\": rpc error: code = NotFound desc = could not find container \"a387fcbb59d24bc9efcabbfb8168ef8be212d3dc0243a7dc7b5011d6a91cf96f\": container with ID starting with a387fcbb59d24bc9efcabbfb8168ef8be212d3dc0243a7dc7b5011d6a91cf96f not found: ID does not exist"
Sep 09 04:15:43.381642 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:43.381620    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"]
Sep 09 04:15:43.385851 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:43.385832    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f0f5e1c0-59959f6bd9-whmlq"]
Sep 09 04:15:44.565171 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:44.565141    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:15:44.780313 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:44.780285    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:44.780313 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:44.780323    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:44.792291 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:44.792270    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:44.989121 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:44.989084    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="597197a3-059f-46d2-957b-40c688ae95ff" path="/var/lib/kubelet/pods/597197a3-059f-46d2-957b-40c688ae95ff/volumes"
Sep 09 04:15:45.388843 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:45.388766    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:15:48.859663 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:48.859627    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:48.912710 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:48.912683    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:49.501530 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:49.501498    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:49.602344 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:49.602312    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:49.702511 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:49.702484    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:52.789540 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:52.789507    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:55.143776 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.143740    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"]
Sep 09 04:15:55.158513 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.158485    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:55.261122 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.261095    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw"]
Sep 09 04:15:55.261668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.261649    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="597197a3-059f-46d2-957b-40c688ae95ff" containerName="payload-processing"
Sep 09 04:15:55.261668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.261669    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="597197a3-059f-46d2-957b-40c688ae95ff" containerName="payload-processing"
Sep 09 04:15:55.261843 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.261786    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="597197a3-059f-46d2-957b-40c688ae95ff" containerName="payload-processing"
Sep 09 04:15:55.264728 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.264710    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw"
Sep 09 04:15:55.267027 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.267002    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:15:55.267136 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.267008    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:15:55.271714 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.271695    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw"]
Sep 09 04:15:55.349426 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.349402    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40138d72-c37e-4d7c-88d5-66f0a5665b8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b80654-4ctdw\" (UID: \"40138d72-c37e-4d7c-88d5-66f0a5665b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw"
Sep 09 04:15:55.450245 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.450192    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40138d72-c37e-4d7c-88d5-66f0a5665b8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b80654-4ctdw\" (UID: \"40138d72-c37e-4d7c-88d5-66f0a5665b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw"
Sep 09 04:15:55.450782 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.450764    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40138d72-c37e-4d7c-88d5-66f0a5665b8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b80654-4ctdw\" (UID: \"40138d72-c37e-4d7c-88d5-66f0a5665b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw"
Sep 09 04:15:55.578476 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.578432    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw"
Sep 09 04:15:55.586226 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.586204    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:55.853141 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:55.853115    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw"]
Sep 09 04:15:55.855108 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:15:55.855083    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40138d72_c37e_4d7c_88d5_66f0a5665b8f.slice/crio-f76c026a69165d1017369abc3700fdbf1024fd7ca46ea228e82e1ffbd9f0f79b WatchSource:0}: Error finding container f76c026a69165d1017369abc3700fdbf1024fd7ca46ea228e82e1ffbd9f0f79b: Status 404 returned error can't find the container with id f76c026a69165d1017369abc3700fdbf1024fd7ca46ea228e82e1ffbd9f0f79b
Sep 09 04:15:56.439707 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:56.439636    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw" event={"ID":"40138d72-c37e-4d7c-88d5-66f0a5665b8f","Type":"ContainerStarted","Data":"ef4a889a3cd0e95ff4976e5111833b9ee334b35d5c292276e25da06b9a6dafb1"}
Sep 09 04:15:56.440033 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:56.439710    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw" event={"ID":"40138d72-c37e-4d7c-88d5-66f0a5665b8f","Type":"ContainerStarted","Data":"f76c026a69165d1017369abc3700fdbf1024fd7ca46ea228e82e1ffbd9f0f79b"}
Sep 09 04:15:56.455334 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:56.455289    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw" podStartSLOduration=1.455275087 podStartE2EDuration="1.455275087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:15:56.453675821 +0000 UTC m=+1055.980046531" watchObservedRunningTime="2026-09-09 04:15:56.455275087 +0000 UTC m=+1055.981645822"
Sep 09 04:15:56.896796 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:56.896768    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:56.996868 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:56.996839    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:15:57.446384 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:57.446347    2553 generic.go:356] "Generic (PLEG): container finished" podID="40138d72-c37e-4d7c-88d5-66f0a5665b8f" containerID="ef4a889a3cd0e95ff4976e5111833b9ee334b35d5c292276e25da06b9a6dafb1" exitCode=0
Sep 09 04:15:57.446839 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:57.446458    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw" event={"ID":"40138d72-c37e-4d7c-88d5-66f0a5665b8f","Type":"ContainerDied","Data":"ef4a889a3cd0e95ff4976e5111833b9ee334b35d5c292276e25da06b9a6dafb1"}
Sep 09 04:15:58.590731 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:58.590705    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw"
Sep 09 04:15:58.681673 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:58.681650    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40138d72-c37e-4d7c-88d5-66f0a5665b8f-maas-api-service-ca\") pod \"40138d72-c37e-4d7c-88d5-66f0a5665b8f\" (UID: \"40138d72-c37e-4d7c-88d5-66f0a5665b8f\") "
Sep 09 04:15:58.681988 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:58.681967    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40138d72-c37e-4d7c-88d5-66f0a5665b8f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "40138d72-c37e-4d7c-88d5-66f0a5665b8f" (UID: "40138d72-c37e-4d7c-88d5-66f0a5665b8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:15:58.782775 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:58.782754    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40138d72-c37e-4d7c-88d5-66f0a5665b8f-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:15:59.458627 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:59.458561    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw"
Sep 09 04:15:59.458757 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:59.458559    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw" event={"ID":"40138d72-c37e-4d7c-88d5-66f0a5665b8f","Type":"ContainerDied","Data":"f76c026a69165d1017369abc3700fdbf1024fd7ca46ea228e82e1ffbd9f0f79b"}
Sep 09 04:15:59.458757 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:15:59.458676    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f76c026a69165d1017369abc3700fdbf1024fd7ca46ea228e82e1ffbd9f0f79b"
Sep 09 04:16:03.008773 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:03.008739    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:03.068879 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:03.068848    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:03.909549 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:03.909511    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:04.370918 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:04.370884    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:04.538412 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:04.538367    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:04.639546 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:04.639468    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:05.386163 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:05.386132    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:06.096566 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.096534    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:06.692951 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.692897    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:06.703032 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.702996    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b80654-29815455-ffd2x"]
Sep 09 04:16:06.705170 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.705149    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b80654-29815455-ffd2x"]
Sep 09 04:16:06.729335 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.729311    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"]
Sep 09 04:16:06.729562 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.729540    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx" podUID="5847b049-7d25-447c-ac12-09fb4d131372" containerName="payload-processing" containerID="cri-o://f4e1e183a9e8e3e289118948676ea469821221a87d8f87acff626dd5be55c27d" gracePeriod=30
Sep 09 04:16:06.735636 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.735609    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"]
Sep 09 04:16:06.735833 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.735807    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l" podUID="6adaa9f6-1047-429a-a5df-843391fbe386" containerName="payload-pre-processing" containerID="cri-o://d6d37ffdfab2a63db397c41b6920d7d178e6d92e4a15c7effa8404f44a9527eb" gracePeriod=30
Sep 09 04:16:06.790631 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.790604    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:06.890965 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.890935    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:06.897710 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.897649    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw"]
Sep 09 04:16:06.900511 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.900488    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b80654-4ctdw"]
Sep 09 04:16:06.950963 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.950898    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:06.990758 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.990733    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40138d72-c37e-4d7c-88d5-66f0a5665b8f" path="/var/lib/kubelet/pods/40138d72-c37e-4d7c-88d5-66f0a5665b8f/volumes"
Sep 09 04:16:06.991112 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:06.991096    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="669c4bab-d7df-4ff7-9a19-26f7ed3093de" path="/var/lib/kubelet/pods/669c4bab-d7df-4ff7-9a19-26f7ed3093de/volumes"
Sep 09 04:16:07.011849 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.011826    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"
Sep 09 04:16:07.014164 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.014143    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"
Sep 09 04:16:07.155454 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.155422    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5847b049-7d25-447c-ac12-09fb4d131372-plugins-config-volume\") pod \"5847b049-7d25-447c-ac12-09fb4d131372\" (UID: \"5847b049-7d25-447c-ac12-09fb4d131372\") "
Sep 09 04:16:07.155636 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.155482    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrz9d\" (UniqueName: \"kubernetes.io/projected/5847b049-7d25-447c-ac12-09fb4d131372-kube-api-access-jrz9d\") pod \"5847b049-7d25-447c-ac12-09fb4d131372\" (UID: \"5847b049-7d25-447c-ac12-09fb4d131372\") "
Sep 09 04:16:07.155636 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.155511    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adaa9f6-1047-429a-a5df-843391fbe386-plugins-config-volume\") pod \"6adaa9f6-1047-429a-a5df-843391fbe386\" (UID: \"6adaa9f6-1047-429a-a5df-843391fbe386\") "
Sep 09 04:16:07.155636 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.155568    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75fk6\" (UniqueName: \"kubernetes.io/projected/6adaa9f6-1047-429a-a5df-843391fbe386-kube-api-access-75fk6\") pod \"6adaa9f6-1047-429a-a5df-843391fbe386\" (UID: \"6adaa9f6-1047-429a-a5df-843391fbe386\") "
Sep 09 04:16:07.155865 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.155766    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5847b049-7d25-447c-ac12-09fb4d131372-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5847b049-7d25-447c-ac12-09fb4d131372" (UID: "5847b049-7d25-447c-ac12-09fb4d131372"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:16:07.155927 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.155901    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6adaa9f6-1047-429a-a5df-843391fbe386-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6adaa9f6-1047-429a-a5df-843391fbe386" (UID: "6adaa9f6-1047-429a-a5df-843391fbe386"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:16:07.157853 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.157827    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5847b049-7d25-447c-ac12-09fb4d131372-kube-api-access-jrz9d" (OuterVolumeSpecName: "kube-api-access-jrz9d") pod "5847b049-7d25-447c-ac12-09fb4d131372" (UID: "5847b049-7d25-447c-ac12-09fb4d131372"). InnerVolumeSpecName "kube-api-access-jrz9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:16:07.157953 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.157923    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6adaa9f6-1047-429a-a5df-843391fbe386-kube-api-access-75fk6" (OuterVolumeSpecName: "kube-api-access-75fk6") pod "6adaa9f6-1047-429a-a5df-843391fbe386" (UID: "6adaa9f6-1047-429a-a5df-843391fbe386"). InnerVolumeSpecName "kube-api-access-75fk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:16:07.256594 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.256562    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5847b049-7d25-447c-ac12-09fb4d131372-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:16:07.256594 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.256590    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrz9d\" (UniqueName: \"kubernetes.io/projected/5847b049-7d25-447c-ac12-09fb4d131372-kube-api-access-jrz9d\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:16:07.256736 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.256603    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adaa9f6-1047-429a-a5df-843391fbe386-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:16:07.256736 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.256617    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75fk6\" (UniqueName: \"kubernetes.io/projected/6adaa9f6-1047-429a-a5df-843391fbe386-kube-api-access-75fk6\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:16:07.273190 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.273158    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:07.504672 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.504644    2553 generic.go:356] "Generic (PLEG): container finished" podID="6adaa9f6-1047-429a-a5df-843391fbe386" containerID="d6d37ffdfab2a63db397c41b6920d7d178e6d92e4a15c7effa8404f44a9527eb" exitCode=0
Sep 09 04:16:07.504773 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.504687    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"
Sep 09 04:16:07.504773 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.504687    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l" event={"ID":"6adaa9f6-1047-429a-a5df-843391fbe386","Type":"ContainerDied","Data":"d6d37ffdfab2a63db397c41b6920d7d178e6d92e4a15c7effa8404f44a9527eb"}
Sep 09 04:16:07.504773 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.504736    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l" event={"ID":"6adaa9f6-1047-429a-a5df-843391fbe386","Type":"ContainerDied","Data":"fda54c4d5dfbdf852d0ec3f17c5b95d8ce0970c5b9b1ca87accc99ff34d3d9c3"}
Sep 09 04:16:07.504773 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.504757    2553 scope.go:117] "RemoveContainer" containerID="d6d37ffdfab2a63db397c41b6920d7d178e6d92e4a15c7effa8404f44a9527eb"
Sep 09 04:16:07.506176 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.506154    2553 generic.go:356] "Generic (PLEG): container finished" podID="5847b049-7d25-447c-ac12-09fb4d131372" containerID="f4e1e183a9e8e3e289118948676ea469821221a87d8f87acff626dd5be55c27d" exitCode=0
Sep 09 04:16:07.506280 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.506208    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"
Sep 09 04:16:07.506460 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.506356    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx" event={"ID":"5847b049-7d25-447c-ac12-09fb4d131372","Type":"ContainerDied","Data":"f4e1e183a9e8e3e289118948676ea469821221a87d8f87acff626dd5be55c27d"}
Sep 09 04:16:07.506512 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.506485    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx" event={"ID":"5847b049-7d25-447c-ac12-09fb4d131372","Type":"ContainerDied","Data":"3ddbea1be2df6199919ec5e9bc20c24eedfed920f008d05bb874d6a9b2728f82"}
Sep 09 04:16:07.515539 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.515520    2553 scope.go:117] "RemoveContainer" containerID="d6d37ffdfab2a63db397c41b6920d7d178e6d92e4a15c7effa8404f44a9527eb"
Sep 09 04:16:07.515895 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:16:07.515878    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d37ffdfab2a63db397c41b6920d7d178e6d92e4a15c7effa8404f44a9527eb\": container with ID starting with d6d37ffdfab2a63db397c41b6920d7d178e6d92e4a15c7effa8404f44a9527eb not found: ID does not exist" containerID="d6d37ffdfab2a63db397c41b6920d7d178e6d92e4a15c7effa8404f44a9527eb"
Sep 09 04:16:07.515950 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.515904    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d37ffdfab2a63db397c41b6920d7d178e6d92e4a15c7effa8404f44a9527eb"} err="failed to get container status \"d6d37ffdfab2a63db397c41b6920d7d178e6d92e4a15c7effa8404f44a9527eb\": rpc error: code = NotFound desc = could not find container \"d6d37ffdfab2a63db397c41b6920d7d178e6d92e4a15c7effa8404f44a9527eb\": container with ID starting with d6d37ffdfab2a63db397c41b6920d7d178e6d92e4a15c7effa8404f44a9527eb not found: ID does not exist"
Sep 09 04:16:07.515950 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.515919    2553 scope.go:117] "RemoveContainer" containerID="f4e1e183a9e8e3e289118948676ea469821221a87d8f87acff626dd5be55c27d"
Sep 09 04:16:07.524025 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.523993    2553 scope.go:117] "RemoveContainer" containerID="f4e1e183a9e8e3e289118948676ea469821221a87d8f87acff626dd5be55c27d"
Sep 09 04:16:07.524309 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:16:07.524289    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e1e183a9e8e3e289118948676ea469821221a87d8f87acff626dd5be55c27d\": container with ID starting with f4e1e183a9e8e3e289118948676ea469821221a87d8f87acff626dd5be55c27d not found: ID does not exist" containerID="f4e1e183a9e8e3e289118948676ea469821221a87d8f87acff626dd5be55c27d"
Sep 09 04:16:07.524359 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.524316    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e1e183a9e8e3e289118948676ea469821221a87d8f87acff626dd5be55c27d"} err="failed to get container status \"f4e1e183a9e8e3e289118948676ea469821221a87d8f87acff626dd5be55c27d\": rpc error: code = NotFound desc = could not find container \"f4e1e183a9e8e3e289118948676ea469821221a87d8f87acff626dd5be55c27d\": container with ID starting with f4e1e183a9e8e3e289118948676ea469821221a87d8f87acff626dd5be55c27d not found: ID does not exist"
Sep 09 04:16:07.527902 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.527881    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"]
Sep 09 04:16:07.531783 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.531764    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b80654-5f87784cfd-k6p4l"]
Sep 09 04:16:07.541850 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.541829    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"]
Sep 09 04:16:07.545731 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.545710    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b80654-7689fbf7b9-gpspx"]
Sep 09 04:16:07.706902 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.706881    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:07.777621 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:07.777556    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:08.866825 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:08.866798    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:08.967247 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:08.967214    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:08.989158 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:08.989132    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5847b049-7d25-447c-ac12-09fb4d131372" path="/var/lib/kubelet/pods/5847b049-7d25-447c-ac12-09fb4d131372/volumes"
Sep 09 04:16:08.989566 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:08.989550    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6adaa9f6-1047-429a-a5df-843391fbe386" path="/var/lib/kubelet/pods/6adaa9f6-1047-429a-a5df-843391fbe386/volumes"
Sep 09 04:16:09.187712 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.187649    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:09.354985 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.354957    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:09.459028 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.458959    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"]
Sep 09 04:16:09.459468 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.459433    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5847b049-7d25-447c-ac12-09fb4d131372" containerName="payload-processing"
Sep 09 04:16:09.459523 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.459471    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5847b049-7d25-447c-ac12-09fb4d131372" containerName="payload-processing"
Sep 09 04:16:09.459523 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.459494    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6adaa9f6-1047-429a-a5df-843391fbe386" containerName="payload-pre-processing"
Sep 09 04:16:09.459523 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.459499    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6adaa9f6-1047-429a-a5df-843391fbe386" containerName="payload-pre-processing"
Sep 09 04:16:09.459523 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.459507    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40138d72-c37e-4d7c-88d5-66f0a5665b8f" containerName="revoke-keys"
Sep 09 04:16:09.459523 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.459513    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="40138d72-c37e-4d7c-88d5-66f0a5665b8f" containerName="revoke-keys"
Sep 09 04:16:09.459672 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.459585    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5847b049-7d25-447c-ac12-09fb4d131372" containerName="payload-processing"
Sep 09 04:16:09.459672 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.459594    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6adaa9f6-1047-429a-a5df-843391fbe386" containerName="payload-pre-processing"
Sep 09 04:16:09.459672 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.459602    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="40138d72-c37e-4d7c-88d5-66f0a5665b8f" containerName="revoke-keys"
Sep 09 04:16:09.464165 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.464150    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"
Sep 09 04:16:09.466584 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.466564    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-46ddac\""
Sep 09 04:16:09.466716 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.466601    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-46ddac-dockercfg-2z9nl\""
Sep 09 04:16:09.472012 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.471990    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"]
Sep 09 04:16:09.576219 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.576186    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrdzz\" (UniqueName: \"kubernetes.io/projected/5e4bda94-fbc9-476f-8346-6f19a253b852-kube-api-access-nrdzz\") pod \"payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb\" (UID: \"5e4bda94-fbc9-476f-8346-6f19a253b852\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"
Sep 09 04:16:09.576379 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.576226    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4bda94-fbc9-476f-8346-6f19a253b852-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb\" (UID: \"5e4bda94-fbc9-476f-8346-6f19a253b852\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"
Sep 09 04:16:09.625935 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.625907    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:09.677248 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.677225    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdzz\" (UniqueName: \"kubernetes.io/projected/5e4bda94-fbc9-476f-8346-6f19a253b852-kube-api-access-nrdzz\") pod \"payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb\" (UID: \"5e4bda94-fbc9-476f-8346-6f19a253b852\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"
Sep 09 04:16:09.677400 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.677269    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4bda94-fbc9-476f-8346-6f19a253b852-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb\" (UID: \"5e4bda94-fbc9-476f-8346-6f19a253b852\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"
Sep 09 04:16:09.677957 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.677938    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4bda94-fbc9-476f-8346-6f19a253b852-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb\" (UID: \"5e4bda94-fbc9-476f-8346-6f19a253b852\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"
Sep 09 04:16:09.685082 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.685062    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrdzz\" (UniqueName: \"kubernetes.io/projected/5e4bda94-fbc9-476f-8346-6f19a253b852-kube-api-access-nrdzz\") pod \"payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb\" (UID: \"5e4bda94-fbc9-476f-8346-6f19a253b852\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"
Sep 09 04:16:09.776691 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:09.776672    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"
Sep 09 04:16:10.050827 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.050800    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"]
Sep 09 04:16:10.052930 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:16:10.052903    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4bda94_fbc9_476f_8346_6f19a253b852.slice/crio-7b7175349b3b6e4279608fbc66b71484ad0a50ac9b407d5a91b6f6c884e240aa WatchSource:0}: Error finding container 7b7175349b3b6e4279608fbc66b71484ad0a50ac9b407d5a91b6f6c884e240aa: Status 404 returned error can't find the container with id 7b7175349b3b6e4279608fbc66b71484ad0a50ac9b407d5a91b6f6c884e240aa
Sep 09 04:16:10.155671 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.155638    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:16:10.168828 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.168803    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw" podUID="65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba" containerName="main" containerID="cri-o://5a89e0a1425ab74eefad2d3d19122d8afb69906e458b898ab1730f028d596969" gracePeriod=45
Sep 09 04:16:10.406138 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.406105    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:16:10.484130 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.484104    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-home\") pod \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") "
Sep 09 04:16:10.484281 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.484190    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-tls-certs\") pod \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") "
Sep 09 04:16:10.484281 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.484216    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-model-cache\") pod \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") "
Sep 09 04:16:10.484281 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.484258    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76pfk\" (UniqueName: \"kubernetes.io/projected/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-kube-api-access-76pfk\") pod \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") "
Sep 09 04:16:10.484481 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.484377    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-tmp-dir\") pod \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") "
Sep 09 04:16:10.484481 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.484374    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-home" (OuterVolumeSpecName: "home") pod "65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba" (UID: "65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:16:10.484593 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.484479    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-dshm\") pod \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\" (UID: \"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba\") "
Sep 09 04:16:10.484593 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.484495    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba" (UID: "65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:16:10.484678 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.484598    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba" (UID: "65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:16:10.485075 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.485052    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:16:10.485185 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.485078    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:16:10.485185 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.485093    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:16:10.486602 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.486577    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba" (UID: "65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:16:10.486686 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.486600    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-dshm" (OuterVolumeSpecName: "dshm") pod "65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba" (UID: "65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:16:10.486686 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.486616    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-kube-api-access-76pfk" (OuterVolumeSpecName: "kube-api-access-76pfk") pod "65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba" (UID: "65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba"). InnerVolumeSpecName "kube-api-access-76pfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:16:10.525969 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.525935    2553 generic.go:356] "Generic (PLEG): container finished" podID="65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba" containerID="5a89e0a1425ab74eefad2d3d19122d8afb69906e458b898ab1730f028d596969" exitCode=0
Sep 09 04:16:10.526075 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.525972    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"
Sep 09 04:16:10.526075 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.526013    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw" event={"ID":"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba","Type":"ContainerDied","Data":"5a89e0a1425ab74eefad2d3d19122d8afb69906e458b898ab1730f028d596969"}
Sep 09 04:16:10.526075 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.526052    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw" event={"ID":"65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba","Type":"ContainerDied","Data":"691880e63198c24c1b4d67afa118bd01f251424b1378d60829295031f668d02c"}
Sep 09 04:16:10.526075 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.526069    2553 scope.go:117] "RemoveContainer" containerID="5a89e0a1425ab74eefad2d3d19122d8afb69906e458b898ab1730f028d596969"
Sep 09 04:16:10.527943 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.527921    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb" event={"ID":"5e4bda94-fbc9-476f-8346-6f19a253b852","Type":"ContainerStarted","Data":"a4528e4c61ad0d35c3c781ec3e6856e3db5a634e42874347e77876260e8ba897"}
Sep 09 04:16:10.528045 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.527970    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb" event={"ID":"5e4bda94-fbc9-476f-8346-6f19a253b852","Type":"ContainerStarted","Data":"7b7175349b3b6e4279608fbc66b71484ad0a50ac9b407d5a91b6f6c884e240aa"}
Sep 09 04:16:10.528045 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.528029    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"
Sep 09 04:16:10.541834 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.541814    2553 scope.go:117] "RemoveContainer" containerID="5a89e0a1425ab74eefad2d3d19122d8afb69906e458b898ab1730f028d596969"
Sep 09 04:16:10.542150 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:16:10.542120    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a89e0a1425ab74eefad2d3d19122d8afb69906e458b898ab1730f028d596969\": container with ID starting with 5a89e0a1425ab74eefad2d3d19122d8afb69906e458b898ab1730f028d596969 not found: ID does not exist" containerID="5a89e0a1425ab74eefad2d3d19122d8afb69906e458b898ab1730f028d596969"
Sep 09 04:16:10.542242 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.542155    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a89e0a1425ab74eefad2d3d19122d8afb69906e458b898ab1730f028d596969"} err="failed to get container status \"5a89e0a1425ab74eefad2d3d19122d8afb69906e458b898ab1730f028d596969\": rpc error: code = NotFound desc = could not find container \"5a89e0a1425ab74eefad2d3d19122d8afb69906e458b898ab1730f028d596969\": container with ID starting with 5a89e0a1425ab74eefad2d3d19122d8afb69906e458b898ab1730f028d596969 not found: ID does not exist"
Sep 09 04:16:10.551810 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.551774    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb" podStartSLOduration=1.551762481 podStartE2EDuration="1.551762481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:16:10.550958381 +0000 UTC m=+1070.077329098" watchObservedRunningTime="2026-09-09 04:16:10.551762481 +0000 UTC m=+1070.078133190"
Sep 09 04:16:10.565016 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.564993    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"]
Sep 09 04:16:10.568528 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.568507    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-b80654/explicit-model-b80654-kserve-7c78576c79-94pzw"]
Sep 09 04:16:10.586130 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.586078    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76pfk\" (UniqueName: \"kubernetes.io/projected/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-kube-api-access-76pfk\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:16:10.586130 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.586098    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:16:10.586130 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.586109    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:16:10.989216 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:10.989184    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba" path="/var/lib/kubelet/pods/65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba/volumes"
Sep 09 04:16:13.667398 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:13.667294    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:13.767606 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:13.767572    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:13.904691 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:13.904664    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:14.321307 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:14.321271    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:14.619870 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:14.619799    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:14.820681 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:14.820637    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:16.164727 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:16.164692    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:16.266575 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:16.266552    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:16.368695 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:16.368672    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:21.536915 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:21.536892    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"
Sep 09 04:16:32.696427 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:32.696394    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:33.015590 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:33.015557    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:33.332682 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:33.332608    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:33.845364 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:33.845332    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:34.022121 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:34.022083    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:34.468656 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:34.468626    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:16:36.092191 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:36.092151    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:36.183728 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:36.183699    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:37.302557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:37.302522    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:37.403357 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:37.403321    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:37.521161 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:37.521136    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:37.821234 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:37.821201    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:37.922381 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:37.922349    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:37.957978 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:37.957953    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"]
Sep 09 04:16:37.958502 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:37.958486    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba" containerName="main"
Sep 09 04:16:37.958581 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:37.958504    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba" containerName="main"
Sep 09 04:16:37.958623 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:37.958589    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="65bbd4fc-ec6d-4c5d-ae91-c01f8acc32ba" containerName="main"
Sep 09 04:16:37.963272 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:37.963256    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"
Sep 09 04:16:37.965637 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:37.965614    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-9f2be5-dockercfg-zs8qq\""
Sep 09 04:16:37.965746 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:37.965625    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-9f2be5\""
Sep 09 04:16:37.971796 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:37.971774    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"]
Sep 09 04:16:38.019150 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:38.019121    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393d7150-22f6-48a0-a718-c590930bea06-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59\" (UID: \"393d7150-22f6-48a0-a718-c590930bea06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"
Sep 09 04:16:38.019338 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:38.019316    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26c7n\" (UniqueName: \"kubernetes.io/projected/393d7150-22f6-48a0-a718-c590930bea06-kube-api-access-26c7n\") pod \"payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59\" (UID: \"393d7150-22f6-48a0-a718-c590930bea06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"
Sep 09 04:16:38.120510 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:38.120412    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393d7150-22f6-48a0-a718-c590930bea06-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59\" (UID: \"393d7150-22f6-48a0-a718-c590930bea06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"
Sep 09 04:16:38.120653 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:38.120616    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26c7n\" (UniqueName: \"kubernetes.io/projected/393d7150-22f6-48a0-a718-c590930bea06-kube-api-access-26c7n\") pod \"payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59\" (UID: \"393d7150-22f6-48a0-a718-c590930bea06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"
Sep 09 04:16:38.121127 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:38.121105    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393d7150-22f6-48a0-a718-c590930bea06-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59\" (UID: \"393d7150-22f6-48a0-a718-c590930bea06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"
Sep 09 04:16:38.129269 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:38.129241    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26c7n\" (UniqueName: \"kubernetes.io/projected/393d7150-22f6-48a0-a718-c590930bea06-kube-api-access-26c7n\") pod \"payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59\" (UID: \"393d7150-22f6-48a0-a718-c590930bea06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"
Sep 09 04:16:38.275045 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:38.275019    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"
Sep 09 04:16:38.546427 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:38.546404    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"]
Sep 09 04:16:38.548639 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:16:38.548614    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393d7150_22f6_48a0_a718_c590930bea06.slice/crio-55fc57c78d324552b4bdcd892a10093252e478dfa5bc80e584bd83c49e996eaa WatchSource:0}: Error finding container 55fc57c78d324552b4bdcd892a10093252e478dfa5bc80e584bd83c49e996eaa: Status 404 returned error can't find the container with id 55fc57c78d324552b4bdcd892a10093252e478dfa5bc80e584bd83c49e996eaa
Sep 09 04:16:38.688574 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:38.688539    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59" event={"ID":"393d7150-22f6-48a0-a718-c590930bea06","Type":"ContainerStarted","Data":"ca6513ad4b8ed1a53ba4ec5fe470f8b2736dcf1d5e76ed8fab0154d8f543841b"}
Sep 09 04:16:38.688719 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:38.688600    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59" event={"ID":"393d7150-22f6-48a0-a718-c590930bea06","Type":"ContainerStarted","Data":"55fc57c78d324552b4bdcd892a10093252e478dfa5bc80e584bd83c49e996eaa"}
Sep 09 04:16:38.688719 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:38.688651    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"
Sep 09 04:16:38.707045 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:38.706995    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59" podStartSLOduration=1.7069788689999998 podStartE2EDuration="1.706978869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:16:38.70440723 +0000 UTC m=+1098.230777940" watchObservedRunningTime="2026-09-09 04:16:38.706978869 +0000 UTC m=+1098.233349577"
Sep 09 04:16:45.343564 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.343531    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"]
Sep 09 04:16:45.347497 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.347482    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.349711 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.349690    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-1215c1\"/\"kube-root-ca.crt\""
Sep 09 04:16:45.350595 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.350570    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-1215c1\"/\"openshift-service-ca.crt\""
Sep 09 04:16:45.350782 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.350756    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-1215c1\"/\"noten-model-1215c1-kserve-self-signed-certs\""
Sep 09 04:16:45.350861 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.350793    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-1215c1\"/\"default-dockercfg-wr7nn\""
Sep 09 04:16:45.355376 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.355352    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"]
Sep 09 04:16:45.386795 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.386772    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-dshm\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.386882 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.386805    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmd7b\" (UniqueName: \"kubernetes.io/projected/5844a161-d607-46c7-b912-3bda3547c715-kube-api-access-wmd7b\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.386882 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.386857    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-home\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.386955 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.386887    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-tmp-dir\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.386955 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.386902    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5844a161-d607-46c7-b912-3bda3547c715-tls-certs\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.386955 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.386953    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-model-cache\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.488027 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.487998    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-model-cache\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.488135 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.488043    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-dshm\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.488189 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.488168    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmd7b\" (UniqueName: \"kubernetes.io/projected/5844a161-d607-46c7-b912-3bda3547c715-kube-api-access-wmd7b\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.488269 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.488253    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-home\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.488326 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.488297    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-tmp-dir\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.488326 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.488317    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5844a161-d607-46c7-b912-3bda3547c715-tls-certs\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.488466 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.488351    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-model-cache\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.488632 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.488609    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-home\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.488724 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.488618    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-tmp-dir\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.490396 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.490378    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-dshm\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.490803 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.490786    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5844a161-d607-46c7-b912-3bda3547c715-tls-certs\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.496499 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.496477    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmd7b\" (UniqueName: \"kubernetes.io/projected/5844a161-d607-46c7-b912-3bda3547c715-kube-api-access-wmd7b\") pod \"noten-model-1215c1-kserve-754d886c5-7x89d\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") " pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.657598 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.657533    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:45.660943 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.660876    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:16:45.928703 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:45.928681    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"]
Sep 09 04:16:45.930365 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:16:45.930338    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5844a161_d607_46c7_b912_3bda3547c715.slice/crio-b8df209887788a86135f21c99739be1429b1860585593b8cadb1a487b498bebf WatchSource:0}: Error finding container b8df209887788a86135f21c99739be1429b1860585593b8cadb1a487b498bebf: Status 404 returned error can't find the container with id b8df209887788a86135f21c99739be1429b1860585593b8cadb1a487b498bebf
Sep 09 04:16:46.737049 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:46.737016    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d" event={"ID":"5844a161-d607-46c7-b912-3bda3547c715","Type":"ContainerStarted","Data":"8b092e394155a62e2384125865124f4ecc1dbd58d824f7be716937e7f0cc75bf"}
Sep 09 04:16:46.737049 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:46.737055    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d" event={"ID":"5844a161-d607-46c7-b912-3bda3547c715","Type":"ContainerStarted","Data":"b8df209887788a86135f21c99739be1429b1860585593b8cadb1a487b498bebf"}
Sep 09 04:16:46.754185 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:46.754136    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d" podStartSLOduration=1.7541213450000002 podStartE2EDuration="1.754121345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:16:46.752346006 +0000 UTC m=+1106.278716717" watchObservedRunningTime="2026-09-09 04:16:46.754121345 +0000 UTC m=+1106.280492054"
Sep 09 04:16:49.695836 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:49.695808    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"
Sep 09 04:16:55.658691 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:55.658659    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:55.659026 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:55.658704    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:55.671000 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:55.670976    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:16:55.799745 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:16:55.799725    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:17:00.951089 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:00.951052    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:02.083530 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:02.083503    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:17:17.130518 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.130487    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:17.436254 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.436176    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"]
Sep 09 04:17:17.439640 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.439624    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.442119 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.442097    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-429d71ed-openshift-default-dockercfg-ps8nf\""
Sep 09 04:17:17.448074 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.448054    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:17.454517 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.454496    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"]
Sep 09 04:17:17.470836 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.470815    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-workload-socket\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.470940 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.470841    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-558vx\" (UniqueName: \"kubernetes.io/projected/b7972eed-5e82-4b54-9d41-b36c41787d7d-kube-api-access-558vx\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.470940 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.470862    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-data\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.470940 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.470891    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7972eed-5e82-4b54-9d41-b36c41787d7d-istiod-ca-cert\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.471062 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.471026    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-workload-certs\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.471104 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.471070    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-envoy\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.471140 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.471117    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-token\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.471181 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.471140    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-podinfo\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.471181 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.471174    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-credential-socket\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.571944 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.571898    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-workload-socket\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.571944 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.571947    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-558vx\" (UniqueName: \"kubernetes.io/projected/b7972eed-5e82-4b54-9d41-b36c41787d7d-kube-api-access-558vx\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.572200 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.571984    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-data\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.572200 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.572028    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7972eed-5e82-4b54-9d41-b36c41787d7d-istiod-ca-cert\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.572200 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.572100    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-workload-certs\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.572200 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.572125    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-envoy\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.572200 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.572173    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-token\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.572200 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.572198    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-podinfo\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.572679 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.572223    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-credential-socket\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.572679 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.572283    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-workload-socket\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.572679 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.572349    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-data\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.572832 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.572810    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7972eed-5e82-4b54-9d41-b36c41787d7d-istiod-ca-cert\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.572912 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.572889    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-credential-socket\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.572979 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.572924    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-workload-certs\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.574887 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.574868    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-envoy\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.575080 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.575058    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-podinfo\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.581887 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.581866    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-558vx\" (UniqueName: \"kubernetes.io/projected/b7972eed-5e82-4b54-9d41-b36c41787d7d-kube-api-access-558vx\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.582208 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.582192    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-token\") pod \"e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") " pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.750125 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.750087    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:17.855590 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:17.855559    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:18.031364 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:18.031340    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"]
Sep 09 04:17:18.032791 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:17:18.032767    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7972eed_5e82_4b54_9d41_b36c41787d7d.slice/crio-992fcd257be5ce3c8a208e9f28d1d622226cd72a859596101418114249099556 WatchSource:0}: Error finding container 992fcd257be5ce3c8a208e9f28d1d622226cd72a859596101418114249099556: Status 404 returned error can't find the container with id 992fcd257be5ce3c8a208e9f28d1d622226cd72a859596101418114249099556
Sep 09 04:17:18.035216 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:18.035178    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:17:18.035315 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:18.035259    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:17:18.035315 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:18.035300    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:17:18.893466 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:18.893419    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:18.919831 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:18.919804    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp" event={"ID":"b7972eed-5e82-4b54-9d41-b36c41787d7d","Type":"ContainerStarted","Data":"749df95076e0335bb9fed5bef6a3b9d68a17ee856c18d4c71b1726793f54848c"}
Sep 09 04:17:18.919983 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:18.919857    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp" event={"ID":"b7972eed-5e82-4b54-9d41-b36c41787d7d","Type":"ContainerStarted","Data":"992fcd257be5ce3c8a208e9f28d1d622226cd72a859596101418114249099556"}
Sep 09 04:17:18.942898 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:18.942850    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp" podStartSLOduration=1.942835884 podStartE2EDuration="1.942835884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:17:18.940859126 +0000 UTC m=+1138.467229846" watchObservedRunningTime="2026-09-09 04:17:18.942835884 +0000 UTC m=+1138.469206592"
Sep 09 04:17:19.138792 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:19.138760    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:19.397188 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:19.397154    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:19.750944 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:19.750913    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:19.755680 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:19.755656    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:19.926611 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:19.926583    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:19.927559 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:19.927541    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:21.006343 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:21.006312    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:21.107850 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:21.107816    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:22.315352 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:22.315319    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:22.428696 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:22.428665    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:22.757854 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:22.757822    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:23.023517 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.023424    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:23.115339 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.115301    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:23.182731 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.182704    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"]
Sep 09 04:17:23.186610 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.186589    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"
Sep 09 04:17:23.191132 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.191111    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-429d71ed\""
Sep 09 04:17:23.201032 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.201009    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"]
Sep 09 04:17:23.225294 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.225260    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9djs\" (UniqueName: \"kubernetes.io/projected/4f26135d-6ca6-47bc-b61e-c65d3c24ac42-kube-api-access-q9djs\") pod \"payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl\" (UID: \"4f26135d-6ca6-47bc-b61e-c65d3c24ac42\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"
Sep 09 04:17:23.225605 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.225574    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f26135d-6ca6-47bc-b61e-c65d3c24ac42-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl\" (UID: \"4f26135d-6ca6-47bc-b61e-c65d3c24ac42\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"
Sep 09 04:17:23.327119 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.327033    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f26135d-6ca6-47bc-b61e-c65d3c24ac42-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl\" (UID: \"4f26135d-6ca6-47bc-b61e-c65d3c24ac42\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"
Sep 09 04:17:23.327119 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.327093    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9djs\" (UniqueName: \"kubernetes.io/projected/4f26135d-6ca6-47bc-b61e-c65d3c24ac42-kube-api-access-q9djs\") pod \"payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl\" (UID: \"4f26135d-6ca6-47bc-b61e-c65d3c24ac42\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"
Sep 09 04:17:23.327827 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.327809    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f26135d-6ca6-47bc-b61e-c65d3c24ac42-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl\" (UID: \"4f26135d-6ca6-47bc-b61e-c65d3c24ac42\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"
Sep 09 04:17:23.337156 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.337131    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9djs\" (UniqueName: \"kubernetes.io/projected/4f26135d-6ca6-47bc-b61e-c65d3c24ac42-kube-api-access-q9djs\") pod \"payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl\" (UID: \"4f26135d-6ca6-47bc-b61e-c65d3c24ac42\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"
Sep 09 04:17:23.497458 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.497408    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"
Sep 09 04:17:23.547336 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.547283    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:23.776111 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.776081    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"]
Sep 09 04:17:23.779195 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:17:23.779167    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f26135d_6ca6_47bc_b61e_c65d3c24ac42.slice/crio-b066c42ef799c420a0a0795d536f9197fa23f933fb51f35af88bb39dbb09dbe1 WatchSource:0}: Error finding container b066c42ef799c420a0a0795d536f9197fa23f933fb51f35af88bb39dbb09dbe1: Status 404 returned error can't find the container with id b066c42ef799c420a0a0795d536f9197fa23f933fb51f35af88bb39dbb09dbe1
Sep 09 04:17:23.952415 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.952383    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl" event={"ID":"4f26135d-6ca6-47bc-b61e-c65d3c24ac42","Type":"ContainerStarted","Data":"14075c702e2e231c12402b5a6e6a3849e511ff56939fc2d35059a4e437b48139"}
Sep 09 04:17:23.952572 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.952457    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl" event={"ID":"4f26135d-6ca6-47bc-b61e-c65d3c24ac42","Type":"ContainerStarted","Data":"b066c42ef799c420a0a0795d536f9197fa23f933fb51f35af88bb39dbb09dbe1"}
Sep 09 04:17:23.952614 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.952584    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"
Sep 09 04:17:23.973228 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:23.973187    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl" podStartSLOduration=0.973173289 podStartE2EDuration="973.173289ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:17:23.971356672 +0000 UTC m=+1143.497727383" watchObservedRunningTime="2026-09-09 04:17:23.973173289 +0000 UTC m=+1143.499543998"
Sep 09 04:17:24.224525 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:24.224499    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"]
Sep 09 04:17:24.298641 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:24.298610    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:24.414417 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:24.414386    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:26.670925 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:26.670892    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:17:34.962148 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:34.962114    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"
Sep 09 04:17:35.068350 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:35.068305    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:35.642709 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:35.642680    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb"]
Sep 09 04:17:35.646280 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:35.646265    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb"
Sep 09 04:17:35.648755 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:35.648737    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:17:35.648846 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:35.648778    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:17:35.654732 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:35.654395    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb"]
Sep 09 04:17:35.732356 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:35.732332    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3e1e1c-84f9-42db-a706-9744bc1135a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb\" (UID: \"1f3e1e1c-84f9-42db-a706-9744bc1135a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb"
Sep 09 04:17:35.786672 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:35.786644    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:35.833563 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:35.833541    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3e1e1c-84f9-42db-a706-9744bc1135a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb\" (UID: \"1f3e1e1c-84f9-42db-a706-9744bc1135a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb"
Sep 09 04:17:35.834113 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:35.834091    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3e1e1c-84f9-42db-a706-9744bc1135a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb\" (UID: \"1f3e1e1c-84f9-42db-a706-9744bc1135a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb"
Sep 09 04:17:35.958048 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:35.957981    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb"
Sep 09 04:17:36.230310 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:36.230291    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb"]
Sep 09 04:17:36.232193 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:17:36.232169    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3e1e1c_84f9_42db_a706_9744bc1135a2.slice/crio-918b15fe73d42dfa0706ed14ff9d1e68595bce89122dec88e88b4c9e211f3b6e WatchSource:0}: Error finding container 918b15fe73d42dfa0706ed14ff9d1e68595bce89122dec88e88b4c9e211f3b6e: Status 404 returned error can't find the container with id 918b15fe73d42dfa0706ed14ff9d1e68595bce89122dec88e88b4c9e211f3b6e
Sep 09 04:17:37.033414 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:37.033378    2553 generic.go:356] "Generic (PLEG): container finished" podID="1f3e1e1c-84f9-42db-a706-9744bc1135a2" containerID="05ef9c25193f22a82cf7463ba7d595387bfea5e302997f9946036cbcadb5446f" exitCode=0
Sep 09 04:17:37.033562 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:37.033465    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb" event={"ID":"1f3e1e1c-84f9-42db-a706-9744bc1135a2","Type":"ContainerDied","Data":"05ef9c25193f22a82cf7463ba7d595387bfea5e302997f9946036cbcadb5446f"}
Sep 09 04:17:37.033562 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:37.033514    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb" event={"ID":"1f3e1e1c-84f9-42db-a706-9744bc1135a2","Type":"ContainerStarted","Data":"918b15fe73d42dfa0706ed14ff9d1e68595bce89122dec88e88b4c9e211f3b6e"}
Sep 09 04:17:38.173117 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:38.173096    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb"
Sep 09 04:17:38.259431 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:38.259406    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3e1e1c-84f9-42db-a706-9744bc1135a2-maas-api-service-ca\") pod \"1f3e1e1c-84f9-42db-a706-9744bc1135a2\" (UID: \"1f3e1e1c-84f9-42db-a706-9744bc1135a2\") "
Sep 09 04:17:38.260015 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:38.259987    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3e1e1c-84f9-42db-a706-9744bc1135a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f3e1e1c-84f9-42db-a706-9744bc1135a2" (UID: "1f3e1e1c-84f9-42db-a706-9744bc1135a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:17:38.360768 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:38.360705    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3e1e1c-84f9-42db-a706-9744bc1135a2-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:38.571636 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:38.571603    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"]
Sep 09 04:17:38.598391 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:38.598360    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:38.950316 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:38.950283    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:39.045805 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.045781    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb"
Sep 09 04:17:39.045805 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.045789    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb" event={"ID":"1f3e1e1c-84f9-42db-a706-9744bc1135a2","Type":"ContainerDied","Data":"918b15fe73d42dfa0706ed14ff9d1e68595bce89122dec88e88b4c9e211f3b6e"}
Sep 09 04:17:39.045977 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.045836    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="918b15fe73d42dfa0706ed14ff9d1e68595bce89122dec88e88b4c9e211f3b6e"
Sep 09 04:17:39.244403 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.244367    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d" podUID="5844a161-d607-46c7-b912-3bda3547c715" containerName="main" containerID="cri-o://8b092e394155a62e2384125865124f4ecc1dbd58d824f7be716937e7f0cc75bf" gracePeriod=45
Sep 09 04:17:39.489121 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.489101    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:17:39.572130 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.572060    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmd7b\" (UniqueName: \"kubernetes.io/projected/5844a161-d607-46c7-b912-3bda3547c715-kube-api-access-wmd7b\") pod \"5844a161-d607-46c7-b912-3bda3547c715\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") "
Sep 09 04:17:39.572130 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.572123    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-model-cache\") pod \"5844a161-d607-46c7-b912-3bda3547c715\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") "
Sep 09 04:17:39.572300 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.572152    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-home\") pod \"5844a161-d607-46c7-b912-3bda3547c715\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") "
Sep 09 04:17:39.572300 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.572172    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-dshm\") pod \"5844a161-d607-46c7-b912-3bda3547c715\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") "
Sep 09 04:17:39.572300 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.572206    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5844a161-d607-46c7-b912-3bda3547c715-tls-certs\") pod \"5844a161-d607-46c7-b912-3bda3547c715\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") "
Sep 09 04:17:39.572495 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.572344    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-tmp-dir\") pod \"5844a161-d607-46c7-b912-3bda3547c715\" (UID: \"5844a161-d607-46c7-b912-3bda3547c715\") "
Sep 09 04:17:39.572495 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.572393    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-model-cache" (OuterVolumeSpecName: "model-cache") pod "5844a161-d607-46c7-b912-3bda3547c715" (UID: "5844a161-d607-46c7-b912-3bda3547c715"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:39.572495 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.572479    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-home" (OuterVolumeSpecName: "home") pod "5844a161-d607-46c7-b912-3bda3547c715" (UID: "5844a161-d607-46c7-b912-3bda3547c715"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:39.572665 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.572623    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5844a161-d607-46c7-b912-3bda3547c715" (UID: "5844a161-d607-46c7-b912-3bda3547c715"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:39.573555 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.573516    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:39.573671 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.573570    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:39.573671 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.573586    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:39.581506 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.581474    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5844a161-d607-46c7-b912-3bda3547c715-kube-api-access-wmd7b" (OuterVolumeSpecName: "kube-api-access-wmd7b") pod "5844a161-d607-46c7-b912-3bda3547c715" (UID: "5844a161-d607-46c7-b912-3bda3547c715"). InnerVolumeSpecName "kube-api-access-wmd7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:17:39.581897 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.581865    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-dshm" (OuterVolumeSpecName: "dshm") pod "5844a161-d607-46c7-b912-3bda3547c715" (UID: "5844a161-d607-46c7-b912-3bda3547c715"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:39.582023 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.581863    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5844a161-d607-46c7-b912-3bda3547c715-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5844a161-d607-46c7-b912-3bda3547c715" (UID: "5844a161-d607-46c7-b912-3bda3547c715"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:17:39.674598 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.674574    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmd7b\" (UniqueName: \"kubernetes.io/projected/5844a161-d607-46c7-b912-3bda3547c715-kube-api-access-wmd7b\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:39.674598 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.674597    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5844a161-d607-46c7-b912-3bda3547c715-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:39.674729 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.674605    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5844a161-d607-46c7-b912-3bda3547c715-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:39.768566 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:39.768543    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:40.052970 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:40.052942    2553 generic.go:356] "Generic (PLEG): container finished" podID="5844a161-d607-46c7-b912-3bda3547c715" containerID="8b092e394155a62e2384125865124f4ecc1dbd58d824f7be716937e7f0cc75bf" exitCode=0
Sep 09 04:17:40.053164 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:40.053016    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"
Sep 09 04:17:40.053164 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:40.053022    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d" event={"ID":"5844a161-d607-46c7-b912-3bda3547c715","Type":"ContainerDied","Data":"8b092e394155a62e2384125865124f4ecc1dbd58d824f7be716937e7f0cc75bf"}
Sep 09 04:17:40.053164 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:40.053082    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d" event={"ID":"5844a161-d607-46c7-b912-3bda3547c715","Type":"ContainerDied","Data":"b8df209887788a86135f21c99739be1429b1860585593b8cadb1a487b498bebf"}
Sep 09 04:17:40.053164 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:40.053105    2553 scope.go:117] "RemoveContainer" containerID="8b092e394155a62e2384125865124f4ecc1dbd58d824f7be716937e7f0cc75bf"
Sep 09 04:17:40.062900 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:40.062885    2553 scope.go:117] "RemoveContainer" containerID="8b092e394155a62e2384125865124f4ecc1dbd58d824f7be716937e7f0cc75bf"
Sep 09 04:17:40.063169 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:17:40.063151    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b092e394155a62e2384125865124f4ecc1dbd58d824f7be716937e7f0cc75bf\": container with ID starting with 8b092e394155a62e2384125865124f4ecc1dbd58d824f7be716937e7f0cc75bf not found: ID does not exist" containerID="8b092e394155a62e2384125865124f4ecc1dbd58d824f7be716937e7f0cc75bf"
Sep 09 04:17:40.063232 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:40.063181    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b092e394155a62e2384125865124f4ecc1dbd58d824f7be716937e7f0cc75bf"} err="failed to get container status \"8b092e394155a62e2384125865124f4ecc1dbd58d824f7be716937e7f0cc75bf\": rpc error: code = NotFound desc = could not find container \"8b092e394155a62e2384125865124f4ecc1dbd58d824f7be716937e7f0cc75bf\": container with ID starting with 8b092e394155a62e2384125865124f4ecc1dbd58d824f7be716937e7f0cc75bf not found: ID does not exist"
Sep 09 04:17:40.077798 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:40.077772    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"]
Sep 09 04:17:40.082019 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:40.082000    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-1215c1/noten-model-1215c1-kserve-754d886c5-7x89d"]
Sep 09 04:17:40.740780 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:40.740754    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"]
Sep 09 04:17:40.756141 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:40.756110    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:40.989162 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:40.989132    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5844a161-d607-46c7-b912-3bda3547c715" path="/var/lib/kubelet/pods/5844a161-d607-46c7-b912-3bda3547c715/volumes"
Sep 09 04:17:43.546620 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:43.546536    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:46.002095 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:46.002051    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:46.114250 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:46.114192    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:46.171463 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:46.171411    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb"]
Sep 09 04:17:46.176093 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:46.176068    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-359adf-zqwwb"]
Sep 09 04:17:46.242113 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:46.242075    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:46.607568 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:46.607532    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:46.992191 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:46.992160    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f3e1e1c-84f9-42db-a706-9744bc1135a2" path="/var/lib/kubelet/pods/1f3e1e1c-84f9-42db-a706-9744bc1135a2/volumes"
Sep 09 04:17:51.673962 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:51.673927    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:53.592217 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.592172    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2" podUID="e162efbb-6638-4b6a-a051-cc6c95e626ca" containerName="main" containerID="cri-o://4b9e307b798c74bea46aa84d2933147f2bc44f0ca48bd559be7947ceaeaea857" gracePeriod=45
Sep 09 04:17:53.835559 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.835539    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:17:53.916950 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.916891    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-tmp-dir\") pod \"e162efbb-6638-4b6a-a051-cc6c95e626ca\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") "
Sep 09 04:17:53.917095 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.916953    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf792\" (UniqueName: \"kubernetes.io/projected/e162efbb-6638-4b6a-a051-cc6c95e626ca-kube-api-access-pf792\") pod \"e162efbb-6638-4b6a-a051-cc6c95e626ca\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") "
Sep 09 04:17:53.917095 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.916984    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e162efbb-6638-4b6a-a051-cc6c95e626ca-tls-certs\") pod \"e162efbb-6638-4b6a-a051-cc6c95e626ca\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") "
Sep 09 04:17:53.917095 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.917012    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-home\") pod \"e162efbb-6638-4b6a-a051-cc6c95e626ca\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") "
Sep 09 04:17:53.917095 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.917064    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-model-cache\") pod \"e162efbb-6638-4b6a-a051-cc6c95e626ca\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") "
Sep 09 04:17:53.917299 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.917095    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-dshm\") pod \"e162efbb-6638-4b6a-a051-cc6c95e626ca\" (UID: \"e162efbb-6638-4b6a-a051-cc6c95e626ca\") "
Sep 09 04:17:53.917299 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.917165    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e162efbb-6638-4b6a-a051-cc6c95e626ca" (UID: "e162efbb-6638-4b6a-a051-cc6c95e626ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:53.917461 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.917401    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-home" (OuterVolumeSpecName: "home") pod "e162efbb-6638-4b6a-a051-cc6c95e626ca" (UID: "e162efbb-6638-4b6a-a051-cc6c95e626ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:53.917518 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.917406    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "e162efbb-6638-4b6a-a051-cc6c95e626ca" (UID: "e162efbb-6638-4b6a-a051-cc6c95e626ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:53.917563 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.917535    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:53.917563 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.917550    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:53.917563 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.917560    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:53.919364 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.919340    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-dshm" (OuterVolumeSpecName: "dshm") pod "e162efbb-6638-4b6a-a051-cc6c95e626ca" (UID: "e162efbb-6638-4b6a-a051-cc6c95e626ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:53.919675 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.919656    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e162efbb-6638-4b6a-a051-cc6c95e626ca-kube-api-access-pf792" (OuterVolumeSpecName: "kube-api-access-pf792") pod "e162efbb-6638-4b6a-a051-cc6c95e626ca" (UID: "e162efbb-6638-4b6a-a051-cc6c95e626ca"). InnerVolumeSpecName "kube-api-access-pf792". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:17:53.919737 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:53.919659    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e162efbb-6638-4b6a-a051-cc6c95e626ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e162efbb-6638-4b6a-a051-cc6c95e626ca" (UID: "e162efbb-6638-4b6a-a051-cc6c95e626ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:17:54.018217 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.018195    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf792\" (UniqueName: \"kubernetes.io/projected/e162efbb-6638-4b6a-a051-cc6c95e626ca-kube-api-access-pf792\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:54.018217 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.018215    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e162efbb-6638-4b6a-a051-cc6c95e626ca-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:54.018345 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.018225    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e162efbb-6638-4b6a-a051-cc6c95e626ca-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:54.151021 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.150992    2553 generic.go:356] "Generic (PLEG): container finished" podID="e162efbb-6638-4b6a-a051-cc6c95e626ca" containerID="4b9e307b798c74bea46aa84d2933147f2bc44f0ca48bd559be7947ceaeaea857" exitCode=0
Sep 09 04:17:54.151135 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.151042    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"
Sep 09 04:17:54.151202 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.151041    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2" event={"ID":"e162efbb-6638-4b6a-a051-cc6c95e626ca","Type":"ContainerDied","Data":"4b9e307b798c74bea46aa84d2933147f2bc44f0ca48bd559be7947ceaeaea857"}
Sep 09 04:17:54.151252 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.151199    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2" event={"ID":"e162efbb-6638-4b6a-a051-cc6c95e626ca","Type":"ContainerDied","Data":"c636691f6bdacdeba1854e8520432f72ed7c9a8f6c297c85fa4bd2d54be40205"}
Sep 09 04:17:54.151252 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.151220    2553 scope.go:117] "RemoveContainer" containerID="4b9e307b798c74bea46aa84d2933147f2bc44f0ca48bd559be7947ceaeaea857"
Sep 09 04:17:54.152880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.152857    2553 generic.go:356] "Generic (PLEG): container finished" podID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerID="749df95076e0335bb9fed5bef6a3b9d68a17ee856c18d4c71b1726793f54848c" exitCode=0
Sep 09 04:17:54.153005 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.152938    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp" event={"ID":"b7972eed-5e82-4b54-9d41-b36c41787d7d","Type":"ContainerDied","Data":"749df95076e0335bb9fed5bef6a3b9d68a17ee856c18d4c71b1726793f54848c"}
Sep 09 04:17:54.153345 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.153329    2553 scope.go:117] "RemoveContainer" containerID="749df95076e0335bb9fed5bef6a3b9d68a17ee856c18d4c71b1726793f54848c"
Sep 09 04:17:54.154557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.154529    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:17:54.154652 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.154599    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:17:54.154652 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.154631    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 04:17:54.161141 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.161127    2553 scope.go:117] "RemoveContainer" containerID="4b9e307b798c74bea46aa84d2933147f2bc44f0ca48bd559be7947ceaeaea857"
Sep 09 04:17:54.161397 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:17:54.161378    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9e307b798c74bea46aa84d2933147f2bc44f0ca48bd559be7947ceaeaea857\": container with ID starting with 4b9e307b798c74bea46aa84d2933147f2bc44f0ca48bd559be7947ceaeaea857 not found: ID does not exist" containerID="4b9e307b798c74bea46aa84d2933147f2bc44f0ca48bd559be7947ceaeaea857"
Sep 09 04:17:54.161480 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.161408    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9e307b798c74bea46aa84d2933147f2bc44f0ca48bd559be7947ceaeaea857"} err="failed to get container status \"4b9e307b798c74bea46aa84d2933147f2bc44f0ca48bd559be7947ceaeaea857\": rpc error: code = NotFound desc = could not find container \"4b9e307b798c74bea46aa84d2933147f2bc44f0ca48bd559be7947ceaeaea857\": container with ID starting with 4b9e307b798c74bea46aa84d2933147f2bc44f0ca48bd559be7947ceaeaea857 not found: ID does not exist"
Sep 09 04:17:54.191798 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.191775    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"]
Sep 09 04:17:54.195084 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.195063    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-359adf/test-model-359adf-kserve-6dc674c4f7-6g5z2"]
Sep 09 04:17:54.989524 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:54.989495    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e162efbb-6638-4b6a-a051-cc6c95e626ca" path="/var/lib/kubelet/pods/e162efbb-6638-4b6a-a051-cc6c95e626ca/volumes"
Sep 09 04:17:55.168600 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:55.168564    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp" event={"ID":"b7972eed-5e82-4b54-9d41-b36c41787d7d","Type":"ContainerStarted","Data":"c7c89193045fb15c8ea33f5d22e4a459e7906b83c58f755d3548e90baccea318"}
Sep 09 04:17:55.750205 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:55.750173    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:17:55.750537 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:55.750511    2553 patch_prober.go:28] interesting pod/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 09 04:17:55.750659 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:55.750572    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp" podUID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 09 04:17:55.761879 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:55.761845    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp" podUID="cc27e879-ae0c-41b5-872c-84453eb29fed" containerName="main" containerID="cri-o://78b3fb871dfb5a93312a869ccab1bba079167a6801a414e5e64e958ce2b5babe" gracePeriod=45
Sep 09 04:17:56.049032 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.049008    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:17:56.138283 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.138258    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-dshm\") pod \"cc27e879-ae0c-41b5-872c-84453eb29fed\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") "
Sep 09 04:17:56.138481 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.138298    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-model-cache\") pod \"cc27e879-ae0c-41b5-872c-84453eb29fed\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") "
Sep 09 04:17:56.138481 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.138356    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc27e879-ae0c-41b5-872c-84453eb29fed-tls-certs\") pod \"cc27e879-ae0c-41b5-872c-84453eb29fed\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") "
Sep 09 04:17:56.138481 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.138385    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-home\") pod \"cc27e879-ae0c-41b5-872c-84453eb29fed\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") "
Sep 09 04:17:56.138662 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.138597    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc27e879-ae0c-41b5-872c-84453eb29fed" (UID: "cc27e879-ae0c-41b5-872c-84453eb29fed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:56.138662 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.138601    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-tmp-dir\") pod \"cc27e879-ae0c-41b5-872c-84453eb29fed\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") "
Sep 09 04:17:56.138770 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.138686    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-home" (OuterVolumeSpecName: "home") pod "cc27e879-ae0c-41b5-872c-84453eb29fed" (UID: "cc27e879-ae0c-41b5-872c-84453eb29fed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:56.138770 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.138690    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdkxv\" (UniqueName: \"kubernetes.io/projected/cc27e879-ae0c-41b5-872c-84453eb29fed-kube-api-access-pdkxv\") pod \"cc27e879-ae0c-41b5-872c-84453eb29fed\" (UID: \"cc27e879-ae0c-41b5-872c-84453eb29fed\") "
Sep 09 04:17:56.138770 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.138751    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc27e879-ae0c-41b5-872c-84453eb29fed" (UID: "cc27e879-ae0c-41b5-872c-84453eb29fed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:56.139637 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.139576    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:56.139637 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.139603    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:56.139637 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.139616    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:56.140797 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.140775    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-dshm" (OuterVolumeSpecName: "dshm") pod "cc27e879-ae0c-41b5-872c-84453eb29fed" (UID: "cc27e879-ae0c-41b5-872c-84453eb29fed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:56.140906 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.140825    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc27e879-ae0c-41b5-872c-84453eb29fed-kube-api-access-pdkxv" (OuterVolumeSpecName: "kube-api-access-pdkxv") pod "cc27e879-ae0c-41b5-872c-84453eb29fed" (UID: "cc27e879-ae0c-41b5-872c-84453eb29fed"). InnerVolumeSpecName "kube-api-access-pdkxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:17:56.140951 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.140903    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc27e879-ae0c-41b5-872c-84453eb29fed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc27e879-ae0c-41b5-872c-84453eb29fed" (UID: "cc27e879-ae0c-41b5-872c-84453eb29fed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:17:56.175460 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.175418    2553 generic.go:356] "Generic (PLEG): container finished" podID="cc27e879-ae0c-41b5-872c-84453eb29fed" containerID="78b3fb871dfb5a93312a869ccab1bba079167a6801a414e5e64e958ce2b5babe" exitCode=0
Sep 09 04:17:56.175555 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.175483    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"
Sep 09 04:17:56.175555 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.175507    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp" event={"ID":"cc27e879-ae0c-41b5-872c-84453eb29fed","Type":"ContainerDied","Data":"78b3fb871dfb5a93312a869ccab1bba079167a6801a414e5e64e958ce2b5babe"}
Sep 09 04:17:56.175683 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.175553    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp" event={"ID":"cc27e879-ae0c-41b5-872c-84453eb29fed","Type":"ContainerDied","Data":"6c1b86d92dbad471c27a42aec6b211fbc8f8c2bc857d79d2bb9a9bcf3ccf48dc"}
Sep 09 04:17:56.175683 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.175575    2553 scope.go:117] "RemoveContainer" containerID="78b3fb871dfb5a93312a869ccab1bba079167a6801a414e5e64e958ce2b5babe"
Sep 09 04:17:56.187317 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.185435    2553 scope.go:117] "RemoveContainer" containerID="78b3fb871dfb5a93312a869ccab1bba079167a6801a414e5e64e958ce2b5babe"
Sep 09 04:17:56.187914 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:17:56.187892    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b3fb871dfb5a93312a869ccab1bba079167a6801a414e5e64e958ce2b5babe\": container with ID starting with 78b3fb871dfb5a93312a869ccab1bba079167a6801a414e5e64e958ce2b5babe not found: ID does not exist" containerID="78b3fb871dfb5a93312a869ccab1bba079167a6801a414e5e64e958ce2b5babe"
Sep 09 04:17:56.187994 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.187925    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b3fb871dfb5a93312a869ccab1bba079167a6801a414e5e64e958ce2b5babe"} err="failed to get container status \"78b3fb871dfb5a93312a869ccab1bba079167a6801a414e5e64e958ce2b5babe\": rpc error: code = NotFound desc = could not find container \"78b3fb871dfb5a93312a869ccab1bba079167a6801a414e5e64e958ce2b5babe\": container with ID starting with 78b3fb871dfb5a93312a869ccab1bba079167a6801a414e5e64e958ce2b5babe not found: ID does not exist"
Sep 09 04:17:56.220524 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.220493    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"]
Sep 09 04:17:56.221382 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.221355    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-629522/test-model-629522-kserve-7879b99c97-52vvp"]
Sep 09 04:17:56.241154 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.241132    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdkxv\" (UniqueName: \"kubernetes.io/projected/cc27e879-ae0c-41b5-872c-84453eb29fed-kube-api-access-pdkxv\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:56.241154 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.241152    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc27e879-ae0c-41b5-872c-84453eb29fed-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:56.241281 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.241164    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc27e879-ae0c-41b5-872c-84453eb29fed-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:17:56.750426 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.750390    2553 patch_prober.go:28] interesting pod/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 09 04:17:56.750635 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.750482    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp" podUID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 09 04:17:56.989411 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:56.989382    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc27e879-ae0c-41b5-872c-84453eb29fed" path="/var/lib/kubelet/pods/cc27e879-ae0c-41b5-872c-84453eb29fed/volumes"
Sep 09 04:17:57.751414 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:57.751383    2553 patch_prober.go:28] interesting pod/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 09 04:17:57.751866 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:57.751480    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp" podUID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 09 04:17:58.001102 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.001061    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:58.241748 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.241717    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:58.578607 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.578519    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4"]
Sep 09 04:17:58.578972 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.578959    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5844a161-d607-46c7-b912-3bda3547c715" containerName="main"
Sep 09 04:17:58.579023 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.578974    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5844a161-d607-46c7-b912-3bda3547c715" containerName="main"
Sep 09 04:17:58.579023 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.578988    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc27e879-ae0c-41b5-872c-84453eb29fed" containerName="main"
Sep 09 04:17:58.579023 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.578993    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc27e879-ae0c-41b5-872c-84453eb29fed" containerName="main"
Sep 09 04:17:58.579023 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.579000    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e162efbb-6638-4b6a-a051-cc6c95e626ca" containerName="main"
Sep 09 04:17:58.579023 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.579005    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e162efbb-6638-4b6a-a051-cc6c95e626ca" containerName="main"
Sep 09 04:17:58.579023 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.579011    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3e1e1c-84f9-42db-a706-9744bc1135a2" containerName="revoke-keys"
Sep 09 04:17:58.579023 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.579016    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3e1e1c-84f9-42db-a706-9744bc1135a2" containerName="revoke-keys"
Sep 09 04:17:58.579227 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.579084    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e162efbb-6638-4b6a-a051-cc6c95e626ca" containerName="main"
Sep 09 04:17:58.579227 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.579092    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc27e879-ae0c-41b5-872c-84453eb29fed" containerName="main"
Sep 09 04:17:58.579227 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.579100    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f3e1e1c-84f9-42db-a706-9744bc1135a2" containerName="revoke-keys"
Sep 09 04:17:58.579227 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.579107    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5844a161-d607-46c7-b912-3bda3547c715" containerName="main"
Sep 09 04:17:58.581301 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.581277    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4"
Sep 09 04:17:58.583780 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.583762    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:17:58.583878 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.583864    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:17:58.589302 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.589271    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4"]
Sep 09 04:17:58.664613 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.664580    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b58fd2f5-93a7-43fc-b6ed-c9a226c8a682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-629522-24cp4\" (UID: \"b58fd2f5-93a7-43fc-b6ed-c9a226c8a682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4"
Sep 09 04:17:58.690664 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.690637    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:17:58.750783 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.750753    2553 patch_prober.go:28] interesting pod/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 09 04:17:58.750926 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.750805    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp" podUID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 09 04:17:58.765276 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.765246    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b58fd2f5-93a7-43fc-b6ed-c9a226c8a682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-629522-24cp4\" (UID: \"b58fd2f5-93a7-43fc-b6ed-c9a226c8a682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4"
Sep 09 04:17:58.765905 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.765885    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b58fd2f5-93a7-43fc-b6ed-c9a226c8a682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-629522-24cp4\" (UID: \"b58fd2f5-93a7-43fc-b6ed-c9a226c8a682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4"
Sep 09 04:17:58.766540 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.766522    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:17:58.893417 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:58.893354    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4"
Sep 09 04:17:59.178473 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:59.178390    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4"]
Sep 09 04:17:59.180725 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:17:59.180689    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58fd2f5_93a7_43fc_b6ed_c9a226c8a682.slice/crio-e1af8982c3bc4db08d1b068a3a9d98c62e9cfeab409f8b1e44f2f6580b620a1d WatchSource:0}: Error finding container e1af8982c3bc4db08d1b068a3a9d98c62e9cfeab409f8b1e44f2f6580b620a1d: Status 404 returned error can't find the container with id e1af8982c3bc4db08d1b068a3a9d98c62e9cfeab409f8b1e44f2f6580b620a1d
Sep 09 04:17:59.202634 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:59.202587    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4" event={"ID":"b58fd2f5-93a7-43fc-b6ed-c9a226c8a682","Type":"ContainerStarted","Data":"e1af8982c3bc4db08d1b068a3a9d98c62e9cfeab409f8b1e44f2f6580b620a1d"}
Sep 09 04:17:59.751419 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:59.751379    2553 patch_prober.go:28] interesting pod/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 09 04:17:59.751610 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:17:59.751471    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp" podUID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 09 04:18:00.209973 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:00.209943    2553 generic.go:356] "Generic (PLEG): container finished" podID="b58fd2f5-93a7-43fc-b6ed-c9a226c8a682" containerID="4ef940435052504137c11ab05d2f02a310bc9bfb8edcc5c915439bbaa94f8d5b" exitCode=0
Sep 09 04:18:00.210262 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:00.210032    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4" event={"ID":"b58fd2f5-93a7-43fc-b6ed-c9a226c8a682","Type":"ContainerDied","Data":"4ef940435052504137c11ab05d2f02a310bc9bfb8edcc5c915439bbaa94f8d5b"}
Sep 09 04:18:00.754475 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:00.754431    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:18:00.754938 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:00.754918    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:18:00.755691 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:00.755673    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:18:01.347438 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:01.347417    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4"
Sep 09 04:18:01.494991 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:01.494970    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b58fd2f5-93a7-43fc-b6ed-c9a226c8a682-maas-api-service-ca\") pod \"b58fd2f5-93a7-43fc-b6ed-c9a226c8a682\" (UID: \"b58fd2f5-93a7-43fc-b6ed-c9a226c8a682\") "
Sep 09 04:18:01.495279 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:01.495257    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b58fd2f5-93a7-43fc-b6ed-c9a226c8a682-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b58fd2f5-93a7-43fc-b6ed-c9a226c8a682" (UID: "b58fd2f5-93a7-43fc-b6ed-c9a226c8a682"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:18:01.595886 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:01.595862    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b58fd2f5-93a7-43fc-b6ed-c9a226c8a682-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:18:01.729759 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:01.729729    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:02.222967 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:02.222934    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4" event={"ID":"b58fd2f5-93a7-43fc-b6ed-c9a226c8a682","Type":"ContainerDied","Data":"e1af8982c3bc4db08d1b068a3a9d98c62e9cfeab409f8b1e44f2f6580b620a1d"}
Sep 09 04:18:02.222967 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:02.222958    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4"
Sep 09 04:18:02.223178 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:02.222985    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1af8982c3bc4db08d1b068a3a9d98c62e9cfeab409f8b1e44f2f6580b620a1d"
Sep 09 04:18:09.252985 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.252935    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:09.260700 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.260649    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"]
Sep 09 04:18:09.261184 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.261058    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4" podUID="4fecc125-b7a9-407f-a168-14f39568fc47" containerName="payload-pre-processing" containerID="cri-o://15cbf34c9da73e0dfad034f2d8812aa36c410845625a9220f270ecb52b356aae" gracePeriod=30
Sep 09 04:18:09.350601 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.350567    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:09.423793 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.423763    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4"]
Sep 09 04:18:09.425816 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.425798    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-629522-24cp4"]
Sep 09 04:18:09.476814 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.476782    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:09.511048 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.510997    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"
Sep 09 04:18:09.534348 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.534323    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:09.568576 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.568551    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqj26\" (UniqueName: \"kubernetes.io/projected/4fecc125-b7a9-407f-a168-14f39568fc47-kube-api-access-tqj26\") pod \"4fecc125-b7a9-407f-a168-14f39568fc47\" (UID: \"4fecc125-b7a9-407f-a168-14f39568fc47\") "
Sep 09 04:18:09.568686 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.568605    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fecc125-b7a9-407f-a168-14f39568fc47-plugins-config-volume\") pod \"4fecc125-b7a9-407f-a168-14f39568fc47\" (UID: \"4fecc125-b7a9-407f-a168-14f39568fc47\") "
Sep 09 04:18:09.568995 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.568970    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fecc125-b7a9-407f-a168-14f39568fc47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4fecc125-b7a9-407f-a168-14f39568fc47" (UID: "4fecc125-b7a9-407f-a168-14f39568fc47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:18:09.569086 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.569051    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fecc125-b7a9-407f-a168-14f39568fc47-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:18:09.570683 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.570660    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fecc125-b7a9-407f-a168-14f39568fc47-kube-api-access-tqj26" (OuterVolumeSpecName: "kube-api-access-tqj26") pod "4fecc125-b7a9-407f-a168-14f39568fc47" (UID: "4fecc125-b7a9-407f-a168-14f39568fc47"). InnerVolumeSpecName "kube-api-access-tqj26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:18:09.670062 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.670024    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqj26\" (UniqueName: \"kubernetes.io/projected/4fecc125-b7a9-407f-a168-14f39568fc47-kube-api-access-tqj26\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:18:09.799828 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:09.799751    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:10.272227 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:10.272195    2553 generic.go:356] "Generic (PLEG): container finished" podID="4fecc125-b7a9-407f-a168-14f39568fc47" containerID="15cbf34c9da73e0dfad034f2d8812aa36c410845625a9220f270ecb52b356aae" exitCode=0
Sep 09 04:18:10.272646 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:10.272241    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4" event={"ID":"4fecc125-b7a9-407f-a168-14f39568fc47","Type":"ContainerDied","Data":"15cbf34c9da73e0dfad034f2d8812aa36c410845625a9220f270ecb52b356aae"}
Sep 09 04:18:10.272646 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:10.272247    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"
Sep 09 04:18:10.272646 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:10.272302    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4" event={"ID":"4fecc125-b7a9-407f-a168-14f39568fc47","Type":"ContainerDied","Data":"21a01ab4cf6ff115e256b3e52ea8b998cbd2bec6164f4d8a4378dd3698ef9c18"}
Sep 09 04:18:10.272646 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:10.272324    2553 scope.go:117] "RemoveContainer" containerID="15cbf34c9da73e0dfad034f2d8812aa36c410845625a9220f270ecb52b356aae"
Sep 09 04:18:10.281568 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:10.281551    2553 scope.go:117] "RemoveContainer" containerID="15cbf34c9da73e0dfad034f2d8812aa36c410845625a9220f270ecb52b356aae"
Sep 09 04:18:10.281803 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:18:10.281785    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15cbf34c9da73e0dfad034f2d8812aa36c410845625a9220f270ecb52b356aae\": container with ID starting with 15cbf34c9da73e0dfad034f2d8812aa36c410845625a9220f270ecb52b356aae not found: ID does not exist" containerID="15cbf34c9da73e0dfad034f2d8812aa36c410845625a9220f270ecb52b356aae"
Sep 09 04:18:10.281854 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:10.281810    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15cbf34c9da73e0dfad034f2d8812aa36c410845625a9220f270ecb52b356aae"} err="failed to get container status \"15cbf34c9da73e0dfad034f2d8812aa36c410845625a9220f270ecb52b356aae\": rpc error: code = NotFound desc = could not find container \"15cbf34c9da73e0dfad034f2d8812aa36c410845625a9220f270ecb52b356aae\": container with ID starting with 15cbf34c9da73e0dfad034f2d8812aa36c410845625a9220f270ecb52b356aae not found: ID does not exist"
Sep 09 04:18:10.294214 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:10.294189    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"]
Sep 09 04:18:10.298094 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:10.298070    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-629522-59b7f8f854-r7rd4"]
Sep 09 04:18:10.989142 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:10.989111    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fecc125-b7a9-407f-a168-14f39568fc47" path="/var/lib/kubelet/pods/4fecc125-b7a9-407f-a168-14f39568fc47/volumes"
Sep 09 04:18:10.989457 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:10.989429    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b58fd2f5-93a7-43fc-b6ed-c9a226c8a682" path="/var/lib/kubelet/pods/b58fd2f5-93a7-43fc-b6ed-c9a226c8a682/volumes"
Sep 09 04:18:15.259096 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:15.259063    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:18:16.172895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:16.172858    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:17.933260 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:17.933227    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:20.792405 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:20.792368    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:21.062973 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:21.062905    2553 scope.go:117] "RemoveContainer" containerID="7f588ca5cce3fe6e134c975a06771ccf00a7724f356a78ca34e75541940d0d64"
Sep 09 04:18:21.071044 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:21.071027    2553 scope.go:117] "RemoveContainer" containerID="6345fa4ecd2055ce75045dcecf3de1639fd4b208872a64dd8aa75ca301fb5af6"
Sep 09 04:18:21.611214 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:21.611189    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:22.040724 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.040691    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:22.120633 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.120601    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:22.267364 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.267319    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:22.778888 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.778860    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:22.883756 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.883714    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:22.976713 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.976687    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"]
Sep 09 04:18:22.977215 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.977199    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fecc125-b7a9-407f-a168-14f39568fc47" containerName="payload-pre-processing"
Sep 09 04:18:22.977274 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.977219    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fecc125-b7a9-407f-a168-14f39568fc47" containerName="payload-pre-processing"
Sep 09 04:18:22.977274 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.977234    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b58fd2f5-93a7-43fc-b6ed-c9a226c8a682" containerName="revoke-keys"
Sep 09 04:18:22.977274 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.977243    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b58fd2f5-93a7-43fc-b6ed-c9a226c8a682" containerName="revoke-keys"
Sep 09 04:18:22.977412 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.977381    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fecc125-b7a9-407f-a168-14f39568fc47" containerName="payload-pre-processing"
Sep 09 04:18:22.977412 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.977394    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b58fd2f5-93a7-43fc-b6ed-c9a226c8a682" containerName="revoke-keys"
Sep 09 04:18:22.982676 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.982656    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"
Sep 09 04:18:22.985009 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.984991    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-426183-dockercfg-sct77\""
Sep 09 04:18:22.985287 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.985274    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-426183\""
Sep 09 04:18:22.993486 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:22.993426    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"]
Sep 09 04:18:23.095851 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:23.095763    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz\" (UID: \"3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"
Sep 09 04:18:23.096218 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:23.095889    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlx4l\" (UniqueName: \"kubernetes.io/projected/3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c-kube-api-access-hlx4l\") pod \"payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz\" (UID: \"3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"
Sep 09 04:18:23.197154 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:23.197127    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz\" (UID: \"3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"
Sep 09 04:18:23.197310 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:23.197178    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlx4l\" (UniqueName: \"kubernetes.io/projected/3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c-kube-api-access-hlx4l\") pod \"payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz\" (UID: \"3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"
Sep 09 04:18:23.197812 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:23.197793    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz\" (UID: \"3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"
Sep 09 04:18:23.206838 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:23.206812    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlx4l\" (UniqueName: \"kubernetes.io/projected/3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c-kube-api-access-hlx4l\") pod \"payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz\" (UID: \"3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"
Sep 09 04:18:23.294069 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:23.294048    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"
Sep 09 04:18:23.564821 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:23.564795    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"]
Sep 09 04:18:23.566168 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:18:23.566139    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1fe41b_0c5a_4b6e_ac64_fc9bdf45952c.slice/crio-a1acd03affebf44d43a8e14390ea4ad10ed1f72157b5e83d1930f5c52a4cda29 WatchSource:0}: Error finding container a1acd03affebf44d43a8e14390ea4ad10ed1f72157b5e83d1930f5c52a4cda29: Status 404 returned error can't find the container with id a1acd03affebf44d43a8e14390ea4ad10ed1f72157b5e83d1930f5c52a4cda29
Sep 09 04:18:24.355077 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:24.355044    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz" event={"ID":"3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c","Type":"ContainerStarted","Data":"1817eb34a4647c814d525d574df6b7dc107643ce1e9cf604b1e565a2050a0caf"}
Sep 09 04:18:24.355546 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:24.355097    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz" event={"ID":"3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c","Type":"ContainerStarted","Data":"a1acd03affebf44d43a8e14390ea4ad10ed1f72157b5e83d1930f5c52a4cda29"}
Sep 09 04:18:24.355546 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:24.355166    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"
Sep 09 04:18:24.372523 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:24.372482    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz" podStartSLOduration=2.372438707 podStartE2EDuration="2.372438707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:18:24.370925698 +0000 UTC m=+1203.897296407" watchObservedRunningTime="2026-09-09 04:18:24.372438707 +0000 UTC m=+1203.898809417"
Sep 09 04:18:25.004261 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:25.004219    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:30.717312 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:30.717271    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:35.363123 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:35.363092    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"
Sep 09 04:18:43.091394 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.091362    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"]
Sep 09 04:18:43.095351 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.095334    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.098826 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.098803    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-429d71ed\"/\"default-dockercfg-9skpz\""
Sep 09 04:18:43.099674 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.099644    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-429d71ed\"/\"e2e-lifecycle-model-429d71ed-kserve-self-signed-certs\""
Sep 09 04:18:43.099785 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.099701    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-429d71ed\"/\"kube-root-ca.crt\""
Sep 09 04:18:43.099785 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.099745    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-429d71ed\"/\"openshift-service-ca.crt\""
Sep 09 04:18:43.107148 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.107125    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"]
Sep 09 04:18:43.119123 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.119100    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:43.276496 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.276472    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-dshm\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.276600 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.276508    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559c5847-255b-481d-88bc-946ed1b7e057-tls-certs\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.276600 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.276542    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-tmp-dir\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.276698 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.276613    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-model-cache\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.276698 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.276655    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-home\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.276769 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.276715    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njfjn\" (UniqueName: \"kubernetes.io/projected/559c5847-255b-481d-88bc-946ed1b7e057-kube-api-access-njfjn\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.377590 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.377501    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-dshm\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.377590 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.377548    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559c5847-255b-481d-88bc-946ed1b7e057-tls-certs\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.377590 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.377581    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-tmp-dir\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.377880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.377637    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-model-cache\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.377880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.377689    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-home\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.377880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.377855    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njfjn\" (UniqueName: \"kubernetes.io/projected/559c5847-255b-481d-88bc-946ed1b7e057-kube-api-access-njfjn\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.378050 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.377957    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-tmp-dir\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.378050 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.377990    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-home\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.378050 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.378025    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-model-cache\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.379807 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.379785    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-dshm\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.380065 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.380047    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559c5847-255b-481d-88bc-946ed1b7e057-tls-certs\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.387572 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.387553    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njfjn\" (UniqueName: \"kubernetes.io/projected/559c5847-255b-481d-88bc-946ed1b7e057-kube-api-access-njfjn\") pod \"e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") " pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.405206 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.405182    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:43.690570 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:43.690546    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"]
Sep 09 04:18:43.692379 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:18:43.692351    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod559c5847_255b_481d_88bc_946ed1b7e057.slice/crio-24bbb61b0cdb215b5072e0c3dc0c6688f1499aed782747aaf35dd60349f99bc0 WatchSource:0}: Error finding container 24bbb61b0cdb215b5072e0c3dc0c6688f1499aed782747aaf35dd60349f99bc0: Status 404 returned error can't find the container with id 24bbb61b0cdb215b5072e0c3dc0c6688f1499aed782747aaf35dd60349f99bc0
Sep 09 04:18:44.470380 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:44.470348    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz" event={"ID":"559c5847-255b-481d-88bc-946ed1b7e057","Type":"ContainerStarted","Data":"af1a307c76fb3cff9b9ebcacbc31bc8f6dd6a1f2c1873b359b1dc34761d1902e"}
Sep 09 04:18:44.470740 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:44.470390    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz" event={"ID":"559c5847-255b-481d-88bc-946ed1b7e057","Type":"ContainerStarted","Data":"24bbb61b0cdb215b5072e0c3dc0c6688f1499aed782747aaf35dd60349f99bc0"}
Sep 09 04:18:44.687956 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:44.687898    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz" podStartSLOduration=1.6878802130000001 podStartE2EDuration="1.687880213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:18:44.509257378 +0000 UTC m=+1224.035628088" watchObservedRunningTime="2026-09-09 04:18:44.687880213 +0000 UTC m=+1224.214250922"
Sep 09 04:18:44.690204 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:44.690185    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:18:45.727784 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:45.727757    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:46.086185 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:46.086110    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:46.434335 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:46.434260    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:47.882831 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:47.882771    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:49.892730 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:49.892698    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:49.991675 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:49.991641    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:51.089918 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.089883    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:51.191528 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.191503    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:51.249836 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.249808    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:51.590680 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.590651    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:51.651019 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.650993    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:51.703165 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.703132    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"]
Sep 09 04:18:51.707219 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.707197    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"
Sep 09 04:18:51.709572 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.709556    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-9c44b5\""
Sep 09 04:18:51.721607 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.721576    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"]
Sep 09 04:18:51.739504 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.739470    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"]
Sep 09 04:18:51.743306 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.743291    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"
Sep 09 04:18:51.745601 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.745580    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-9c44b5-dockercfg-nc9gj\""
Sep 09 04:18:51.754176 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.754156    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"]
Sep 09 04:18:51.757090 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.757068    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2970d9cb-0b4a-4045-aecb-5e2e34ae8226-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4\" (UID: \"2970d9cb-0b4a-4045-aecb-5e2e34ae8226\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"
Sep 09 04:18:51.757209 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.757105    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8qlx\" (UniqueName: \"kubernetes.io/projected/2970d9cb-0b4a-4045-aecb-5e2e34ae8226-kube-api-access-c8qlx\") pod \"payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4\" (UID: \"2970d9cb-0b4a-4045-aecb-5e2e34ae8226\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"
Sep 09 04:18:51.757279 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.757200    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxwlx\" (UniqueName: \"kubernetes.io/projected/d71f8271-7604-4f45-aa3b-0915614d37a9-kube-api-access-hxwlx\") pod \"payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs\" (UID: \"d71f8271-7604-4f45-aa3b-0915614d37a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"
Sep 09 04:18:51.757364 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.757275    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71f8271-7604-4f45-aa3b-0915614d37a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs\" (UID: \"d71f8271-7604-4f45-aa3b-0915614d37a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"
Sep 09 04:18:51.858471 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.858356    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71f8271-7604-4f45-aa3b-0915614d37a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs\" (UID: \"d71f8271-7604-4f45-aa3b-0915614d37a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"
Sep 09 04:18:51.858620 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.858510    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2970d9cb-0b4a-4045-aecb-5e2e34ae8226-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4\" (UID: \"2970d9cb-0b4a-4045-aecb-5e2e34ae8226\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"
Sep 09 04:18:51.858620 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.858542    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8qlx\" (UniqueName: \"kubernetes.io/projected/2970d9cb-0b4a-4045-aecb-5e2e34ae8226-kube-api-access-c8qlx\") pod \"payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4\" (UID: \"2970d9cb-0b4a-4045-aecb-5e2e34ae8226\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"
Sep 09 04:18:51.858620 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.858608    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxwlx\" (UniqueName: \"kubernetes.io/projected/d71f8271-7604-4f45-aa3b-0915614d37a9-kube-api-access-hxwlx\") pod \"payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs\" (UID: \"d71f8271-7604-4f45-aa3b-0915614d37a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"
Sep 09 04:18:51.859177 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.859160    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71f8271-7604-4f45-aa3b-0915614d37a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs\" (UID: \"d71f8271-7604-4f45-aa3b-0915614d37a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"
Sep 09 04:18:51.859213 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.859164    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2970d9cb-0b4a-4045-aecb-5e2e34ae8226-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4\" (UID: \"2970d9cb-0b4a-4045-aecb-5e2e34ae8226\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"
Sep 09 04:18:51.867416 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.867388    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8qlx\" (UniqueName: \"kubernetes.io/projected/2970d9cb-0b4a-4045-aecb-5e2e34ae8226-kube-api-access-c8qlx\") pod \"payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4\" (UID: \"2970d9cb-0b4a-4045-aecb-5e2e34ae8226\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"
Sep 09 04:18:51.867604 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:51.867589    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxwlx\" (UniqueName: \"kubernetes.io/projected/d71f8271-7604-4f45-aa3b-0915614d37a9-kube-api-access-hxwlx\") pod \"payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs\" (UID: \"d71f8271-7604-4f45-aa3b-0915614d37a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"
Sep 09 04:18:52.024359 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:52.024333    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"
Sep 09 04:18:52.054772 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:52.054739    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"
Sep 09 04:18:52.297091 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:52.297067    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"]
Sep 09 04:18:52.298615 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:18:52.298586    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2970d9cb_0b4a_4045_aecb_5e2e34ae8226.slice/crio-90aafe14ae83716a16f2fb1a28c493205d71d8fecbfb02ced76ebcf4d1cb7645 WatchSource:0}: Error finding container 90aafe14ae83716a16f2fb1a28c493205d71d8fecbfb02ced76ebcf4d1cb7645: Status 404 returned error can't find the container with id 90aafe14ae83716a16f2fb1a28c493205d71d8fecbfb02ced76ebcf4d1cb7645
Sep 09 04:18:52.339979 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:52.339764    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"]
Sep 09 04:18:52.342160 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:18:52.342134    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71f8271_7604_4f45_aa3b_0915614d37a9.slice/crio-414ec6af565035ef10f732873160b54625947a61b33049ed018f0af193cbf9f4 WatchSource:0}: Error finding container 414ec6af565035ef10f732873160b54625947a61b33049ed018f0af193cbf9f4: Status 404 returned error can't find the container with id 414ec6af565035ef10f732873160b54625947a61b33049ed018f0af193cbf9f4
Sep 09 04:18:52.518719 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:52.518593    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs" event={"ID":"d71f8271-7604-4f45-aa3b-0915614d37a9","Type":"ContainerStarted","Data":"37cd8f15e82f7f2a357a8ca2a46e845df058a6f54d61db3f0b5d53d06c5c7c3d"}
Sep 09 04:18:52.518719 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:52.518661    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"
Sep 09 04:18:52.518719 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:52.518686    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs" event={"ID":"d71f8271-7604-4f45-aa3b-0915614d37a9","Type":"ContainerStarted","Data":"414ec6af565035ef10f732873160b54625947a61b33049ed018f0af193cbf9f4"}
Sep 09 04:18:52.520370 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:52.520338    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4" event={"ID":"2970d9cb-0b4a-4045-aecb-5e2e34ae8226","Type":"ContainerStarted","Data":"96128450aa026d6f217c0cf18192c192c15429e2ca3e8242ae088b5d4c81e597"}
Sep 09 04:18:52.520516 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:52.520386    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"
Sep 09 04:18:52.520516 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:52.520403    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4" event={"ID":"2970d9cb-0b4a-4045-aecb-5e2e34ae8226","Type":"ContainerStarted","Data":"90aafe14ae83716a16f2fb1a28c493205d71d8fecbfb02ced76ebcf4d1cb7645"}
Sep 09 04:18:52.538827 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:52.538781    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs" podStartSLOduration=1.5387645829999999 podStartE2EDuration="1.538764583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:18:52.537665401 +0000 UTC m=+1232.064036113" watchObservedRunningTime="2026-09-09 04:18:52.538764583 +0000 UTC m=+1232.065135293"
Sep 09 04:18:52.557617 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:52.557575    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4" podStartSLOduration=1.557562102 podStartE2EDuration="1.557562102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:18:52.555562973 +0000 UTC m=+1232.081933685" watchObservedRunningTime="2026-09-09 04:18:52.557562102 +0000 UTC m=+1232.083932861"
Sep 09 04:18:53.405564 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:53.405520    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:53.405564 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:53.405573    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:53.418245 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:53.418223    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:53.538333 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:53.538306    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:18:58.469020 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.468983    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"]
Sep 09 04:18:58.473171 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.473149    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.475553 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.475535    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-85732c\"/\"unconfigured-w4-85732c-kserve-self-signed-certs\""
Sep 09 04:18:58.481907 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.481888    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"]
Sep 09 04:18:58.522962 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.522941    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-tmp-dir\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.523111 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.523003    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02176de-51f1-4f4a-8b93-e4008228f719-tls-certs\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.523111 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.523035    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bblq\" (UniqueName: \"kubernetes.io/projected/b02176de-51f1-4f4a-8b93-e4008228f719-kube-api-access-9bblq\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.523232 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.523168    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-dshm\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.523232 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.523225    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-home\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.523346 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.523261    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-model-cache\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.624294 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.624265    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-model-cache\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.624430 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.624325    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-tmp-dir\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.624430 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.624375    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02176de-51f1-4f4a-8b93-e4008228f719-tls-certs\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.624430 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.624421    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bblq\" (UniqueName: \"kubernetes.io/projected/b02176de-51f1-4f4a-8b93-e4008228f719-kube-api-access-9bblq\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.624632 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.624537    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-dshm\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.624632 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.624584    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-home\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.624785 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.624765    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-model-cache\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.624847 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.624792    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-tmp-dir\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.624905 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.624890    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-home\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.626898 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.626881    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-dshm\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.627253 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.627233    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02176de-51f1-4f4a-8b93-e4008228f719-tls-certs\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.633602 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.633581    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bblq\" (UniqueName: \"kubernetes.io/projected/b02176de-51f1-4f4a-8b93-e4008228f719-kube-api-access-9bblq\") pod \"unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") " pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:58.751166 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.751132    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:18:58.784256 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:58.784232    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:18:59.063469 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:59.063427    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"]
Sep 09 04:18:59.065901 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:18:59.065875    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb02176de_51f1_4f4a_8b93_e4008228f719.slice/crio-80fbb9da524237830bafed3c84e5050e9df43bd82f6b36cd5ef87b4e88a403b6 WatchSource:0}: Error finding container 80fbb9da524237830bafed3c84e5050e9df43bd82f6b36cd5ef87b4e88a403b6: Status 404 returned error can't find the container with id 80fbb9da524237830bafed3c84e5050e9df43bd82f6b36cd5ef87b4e88a403b6
Sep 09 04:18:59.564079 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:59.564040    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d" event={"ID":"b02176de-51f1-4f4a-8b93-e4008228f719","Type":"ContainerStarted","Data":"8999c189c6961769f6bcb144177cb73156f9e092a0d21ada753ea18a294ab32d"}
Sep 09 04:18:59.564079 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:59.564084    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d" event={"ID":"b02176de-51f1-4f4a-8b93-e4008228f719","Type":"ContainerStarted","Data":"80fbb9da524237830bafed3c84e5050e9df43bd82f6b36cd5ef87b4e88a403b6"}
Sep 09 04:18:59.583132 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:18:59.583080    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d" podStartSLOduration=1.583064327 podStartE2EDuration="1.583064327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:18:59.580959999 +0000 UTC m=+1239.107330710" watchObservedRunningTime="2026-09-09 04:18:59.583064327 +0000 UTC m=+1239.109435036"
Sep 09 04:19:03.528632 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:03.528603    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"
Sep 09 04:19:03.529093 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:03.528821    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"
Sep 09 04:19:07.105377 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:07.105321    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:19:08.785171 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:08.785140    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:19:08.785635 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:08.785185    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:19:08.797995 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:08.797972    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:19:09.635021 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:09.634995    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:19:14.880700 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:14.880624    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:19:16.779914 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:16.779882    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:19:21.112284 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:21.112253    2553 scope.go:117] "RemoveContainer" containerID="a9bcea6f3354987f4a98036d8ca73341462c1629a4da7202e5070b8cbf57104a"
Sep 09 04:19:21.120616 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:21.120595    2553 scope.go:117] "RemoveContainer" containerID="15dcf9faac24b926a42f711121cc12a5c8b873bb1bd5ceeb9d44143f4c095fcd"
Sep 09 04:19:21.128281 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:21.128264    2553 scope.go:117] "RemoveContainer" containerID="72480dd580506a12143d4172ecea54617a7be62c40ca7e32ebcebf7d52512293"
Sep 09 04:19:22.408048 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:22.408009    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:19:22.513104 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:22.513076    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:19:29.673722 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:29.673692    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:19:33.320703 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:33.320675    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d"]
Sep 09 04:19:33.326033 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:33.326015    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d"
Sep 09 04:19:33.333721 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:33.333695    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:19:33.333721 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:33.333706    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:19:33.336256 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:33.336233    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d"]
Sep 09 04:19:33.433723 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:33.433701    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:19:33.456033 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:33.456007    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d\" (UID: \"b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d"
Sep 09 04:19:33.556627 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:33.556602    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d\" (UID: \"b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d"
Sep 09 04:19:33.557203 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:33.557186    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d\" (UID: \"b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d"
Sep 09 04:19:33.637989 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:33.637928    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d"
Sep 09 04:19:33.922031 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:33.922003    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d"]
Sep 09 04:19:33.923928 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:19:33.923902    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ce2ab9_2a91_4912_b642_b1f5a8fcc4e6.slice/crio-2b75a1ec1f4afe61ad0971653a0e9085db4516c5f8af5f67453271e138a7f753 WatchSource:0}: Error finding container 2b75a1ec1f4afe61ad0971653a0e9085db4516c5f8af5f67453271e138a7f753: Status 404 returned error can't find the container with id 2b75a1ec1f4afe61ad0971653a0e9085db4516c5f8af5f67453271e138a7f753
Sep 09 04:19:34.765974 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:34.765945    2553 generic.go:356] "Generic (PLEG): container finished" podID="b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6" containerID="264b108a3e0e62d92209377c80dd672f972558120148d364cf44f5b43f52afb8" exitCode=0
Sep 09 04:19:34.766275 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:34.766035    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d" event={"ID":"b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6","Type":"ContainerDied","Data":"264b108a3e0e62d92209377c80dd672f972558120148d364cf44f5b43f52afb8"}
Sep 09 04:19:34.766275 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:34.766078    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d" event={"ID":"b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6","Type":"ContainerStarted","Data":"2b75a1ec1f4afe61ad0971653a0e9085db4516c5f8af5f67453271e138a7f753"}
Sep 09 04:19:35.906825 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:35.906802    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d"
Sep 09 04:19:35.954097 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:35.954064    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:19:36.079776 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:36.079719    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6-maas-api-service-ca\") pod \"b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6\" (UID: \"b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6\") "
Sep 09 04:19:36.080071 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:36.080047    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6" (UID: "b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:19:36.180712 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:36.180691    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:19:36.779020 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:36.778985    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d" event={"ID":"b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6","Type":"ContainerDied","Data":"2b75a1ec1f4afe61ad0971653a0e9085db4516c5f8af5f67453271e138a7f753"}
Sep 09 04:19:36.779020 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:36.779003    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d"
Sep 09 04:19:36.779264 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:36.779029    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b75a1ec1f4afe61ad0971653a0e9085db4516c5f8af5f67453271e138a7f753"
Sep 09 04:19:48.956031 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:48.956002    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:19:48.992768 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:48.992735    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"]
Sep 09 04:19:48.993085 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:48.993010    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl" podUID="4f26135d-6ca6-47bc-b61e-c65d3c24ac42" containerName="payload-pre-processing" containerID="cri-o://14075c702e2e231c12402b5a6e6a3849e511ff56939fc2d35059a4e437b48139" gracePeriod=30
Sep 09 04:19:49.056191 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.056165    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:19:49.150019 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.149983    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d"]
Sep 09 04:19:49.152313 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.152295    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-429d71ed-ftk4d"]
Sep 09 04:19:49.154931 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.154908    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:19:49.212516 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.212406    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:19:49.247135 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.247111    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"
Sep 09 04:19:49.294574 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.294544    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9djs\" (UniqueName: \"kubernetes.io/projected/4f26135d-6ca6-47bc-b61e-c65d3c24ac42-kube-api-access-q9djs\") pod \"4f26135d-6ca6-47bc-b61e-c65d3c24ac42\" (UID: \"4f26135d-6ca6-47bc-b61e-c65d3c24ac42\") "
Sep 09 04:19:49.294718 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.294645    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f26135d-6ca6-47bc-b61e-c65d3c24ac42-plugins-config-volume\") pod \"4f26135d-6ca6-47bc-b61e-c65d3c24ac42\" (UID: \"4f26135d-6ca6-47bc-b61e-c65d3c24ac42\") "
Sep 09 04:19:49.294983 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.294958    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f26135d-6ca6-47bc-b61e-c65d3c24ac42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f26135d-6ca6-47bc-b61e-c65d3c24ac42" (UID: "4f26135d-6ca6-47bc-b61e-c65d3c24ac42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:19:49.296605 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.296583    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f26135d-6ca6-47bc-b61e-c65d3c24ac42-kube-api-access-q9djs" (OuterVolumeSpecName: "kube-api-access-q9djs") pod "4f26135d-6ca6-47bc-b61e-c65d3c24ac42" (UID: "4f26135d-6ca6-47bc-b61e-c65d3c24ac42"). InnerVolumeSpecName "kube-api-access-q9djs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:19:49.395892 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.395860    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9djs\" (UniqueName: \"kubernetes.io/projected/4f26135d-6ca6-47bc-b61e-c65d3c24ac42-kube-api-access-q9djs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:19:49.395892 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.395889    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f26135d-6ca6-47bc-b61e-c65d3c24ac42-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:19:49.854054 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.854017    2553 generic.go:356] "Generic (PLEG): container finished" podID="4f26135d-6ca6-47bc-b61e-c65d3c24ac42" containerID="14075c702e2e231c12402b5a6e6a3849e511ff56939fc2d35059a4e437b48139" exitCode=0
Sep 09 04:19:49.854249 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.854076    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"
Sep 09 04:19:49.854249 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.854101    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl" event={"ID":"4f26135d-6ca6-47bc-b61e-c65d3c24ac42","Type":"ContainerDied","Data":"14075c702e2e231c12402b5a6e6a3849e511ff56939fc2d35059a4e437b48139"}
Sep 09 04:19:49.854249 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.854179    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl" event={"ID":"4f26135d-6ca6-47bc-b61e-c65d3c24ac42","Type":"ContainerDied","Data":"b066c42ef799c420a0a0795d536f9197fa23f933fb51f35af88bb39dbb09dbe1"}
Sep 09 04:19:49.854249 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.854200    2553 scope.go:117] "RemoveContainer" containerID="14075c702e2e231c12402b5a6e6a3849e511ff56939fc2d35059a4e437b48139"
Sep 09 04:19:49.863913 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.863901    2553 scope.go:117] "RemoveContainer" containerID="14075c702e2e231c12402b5a6e6a3849e511ff56939fc2d35059a4e437b48139"
Sep 09 04:19:49.864183 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:19:49.864165    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14075c702e2e231c12402b5a6e6a3849e511ff56939fc2d35059a4e437b48139\": container with ID starting with 14075c702e2e231c12402b5a6e6a3849e511ff56939fc2d35059a4e437b48139 not found: ID does not exist" containerID="14075c702e2e231c12402b5a6e6a3849e511ff56939fc2d35059a4e437b48139"
Sep 09 04:19:49.864245 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.864193    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14075c702e2e231c12402b5a6e6a3849e511ff56939fc2d35059a4e437b48139"} err="failed to get container status \"14075c702e2e231c12402b5a6e6a3849e511ff56939fc2d35059a4e437b48139\": rpc error: code = NotFound desc = could not find container \"14075c702e2e231c12402b5a6e6a3849e511ff56939fc2d35059a4e437b48139\": container with ID starting with 14075c702e2e231c12402b5a6e6a3849e511ff56939fc2d35059a4e437b48139 not found: ID does not exist"
Sep 09 04:19:49.876125 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.876102    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"]
Sep 09 04:19:49.881969 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:49.881947    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-429d71ed-67656f8498-7j5tl"]
Sep 09 04:19:50.989454 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:50.989420    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f26135d-6ca6-47bc-b61e-c65d3c24ac42" path="/var/lib/kubelet/pods/4f26135d-6ca6-47bc-b61e-c65d3c24ac42/volumes"
Sep 09 04:19:50.989820 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:50.989764    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6" path="/var/lib/kubelet/pods/b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6/volumes"
Sep 09 04:19:52.590629 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:52.590596    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:19:55.960764 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:55.960726    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:19:57.575281 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:57.575244    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"]
Sep 09 04:19:57.671744 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:19:57.671709    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:07.168363 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.168330    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"]
Sep 09 04:20:07.168795 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.168780    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f26135d-6ca6-47bc-b61e-c65d3c24ac42" containerName="payload-pre-processing"
Sep 09 04:20:07.168842 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.168796    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f26135d-6ca6-47bc-b61e-c65d3c24ac42" containerName="payload-pre-processing"
Sep 09 04:20:07.168842 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.168811    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6" containerName="revoke-keys"
Sep 09 04:20:07.168842 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.168816    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6" containerName="revoke-keys"
Sep 09 04:20:07.168939 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.168897    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f26135d-6ca6-47bc-b61e-c65d3c24ac42" containerName="payload-pre-processing"
Sep 09 04:20:07.168939 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.168904    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ce2ab9-2a91-4912-b642-b1f5a8fcc4e6" containerName="revoke-keys"
Sep 09 04:20:07.173419 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.173399    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.175973 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.175954    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-426183\"/\"kube-root-ca.crt\""
Sep 09 04:20:07.176949 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.176921    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-426183\"/\"default-dockercfg-q6z8l\""
Sep 09 04:20:07.177074 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.176961    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-426183\"/\"openshift-service-ca.crt\""
Sep 09 04:20:07.177074 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.177005    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-426183\"/\"auth-test-model-426183-kserve-self-signed-certs\""
Sep 09 04:20:07.181703 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.181665    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"]
Sep 09 04:20:07.196917 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.196895    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:07.249194 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.249165    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-home\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.249299 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.249200    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-model-cache\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.249299 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.249230    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-tmp-dir\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.249299 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.249276    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj5hk\" (UniqueName: \"kubernetes.io/projected/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-kube-api-access-jj5hk\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.249422 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.249386    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-dshm\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.249480 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.249427    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-tls-certs\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.350524 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.350498    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-dshm\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.350700 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.350545    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-tls-certs\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.350700 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.350596    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-home\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.350700 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.350618    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-model-cache\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.350700 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.350656    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-tmp-dir\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.350700 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.350695    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj5hk\" (UniqueName: \"kubernetes.io/projected/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-kube-api-access-jj5hk\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.351011 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.350992    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-model-cache\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.351054 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.351036    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-home\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.351088 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.351059    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-tmp-dir\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.353422 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.353402    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-dshm\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.353682 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.353665    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-tls-certs\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.359874 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.359855    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj5hk\" (UniqueName: \"kubernetes.io/projected/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-kube-api-access-jj5hk\") pod \"auth-test-model-426183-kserve-cb799f6cf-vjnmz\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") " pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.484164 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.484141    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:07.755659 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.755636    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"]
Sep 09 04:20:07.757171 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:20:07.757149    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbaeba7e_5d6f_452b_966a_2e7c1054a17a.slice/crio-b4def1f7e7b620aa5d80b872ce9460bb7f54cd5654f47e8384a5ea28a6232707 WatchSource:0}: Error finding container b4def1f7e7b620aa5d80b872ce9460bb7f54cd5654f47e8384a5ea28a6232707: Status 404 returned error can't find the container with id b4def1f7e7b620aa5d80b872ce9460bb7f54cd5654f47e8384a5ea28a6232707
Sep 09 04:20:07.958598 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.958560    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz" event={"ID":"fbaeba7e-5d6f-452b-966a-2e7c1054a17a","Type":"ContainerStarted","Data":"3456d3a50f8b32b6ad3a2a7c286fd9c54e0286554def5a3a6e3acd8428eabb33"}
Sep 09 04:20:07.958598 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.958603    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz" event={"ID":"fbaeba7e-5d6f-452b-966a-2e7c1054a17a","Type":"ContainerStarted","Data":"b4def1f7e7b620aa5d80b872ce9460bb7f54cd5654f47e8384a5ea28a6232707"}
Sep 09 04:20:07.976029 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:07.975977    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz" podStartSLOduration=0.975963939 podStartE2EDuration="975.963939ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:20:07.974291591 +0000 UTC m=+1307.500662303" watchObservedRunningTime="2026-09-09 04:20:07.975963939 +0000 UTC m=+1307.502334647"
Sep 09 04:20:12.597152 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:12.597111    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz" podUID="559c5847-255b-481d-88bc-946ed1b7e057" containerName="main" containerID="cri-o://af1a307c76fb3cff9b9ebcacbc31bc8f6dd6a1f2c1873b359b1dc34761d1902e" gracePeriod=45
Sep 09 04:20:12.843802 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:12.843781    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:20:12.992275 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:12.992245    2553 generic.go:356] "Generic (PLEG): container finished" podID="559c5847-255b-481d-88bc-946ed1b7e057" containerID="af1a307c76fb3cff9b9ebcacbc31bc8f6dd6a1f2c1873b359b1dc34761d1902e" exitCode=0
Sep 09 04:20:12.992434 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:12.992347    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"
Sep 09 04:20:12.992517 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:12.992497    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz" event={"ID":"559c5847-255b-481d-88bc-946ed1b7e057","Type":"ContainerDied","Data":"af1a307c76fb3cff9b9ebcacbc31bc8f6dd6a1f2c1873b359b1dc34761d1902e"}
Sep 09 04:20:12.992557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:12.992533    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz" event={"ID":"559c5847-255b-481d-88bc-946ed1b7e057","Type":"ContainerDied","Data":"24bbb61b0cdb215b5072e0c3dc0c6688f1499aed782747aaf35dd60349f99bc0"}
Sep 09 04:20:12.992557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:12.992549    2553 scope.go:117] "RemoveContainer" containerID="af1a307c76fb3cff9b9ebcacbc31bc8f6dd6a1f2c1873b359b1dc34761d1902e"
Sep 09 04:20:13.000706 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.000682    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-tmp-dir\") pod \"559c5847-255b-481d-88bc-946ed1b7e057\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") "
Sep 09 04:20:13.000801 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.000728    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559c5847-255b-481d-88bc-946ed1b7e057-tls-certs\") pod \"559c5847-255b-481d-88bc-946ed1b7e057\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") "
Sep 09 04:20:13.000801 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.000795    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njfjn\" (UniqueName: \"kubernetes.io/projected/559c5847-255b-481d-88bc-946ed1b7e057-kube-api-access-njfjn\") pod \"559c5847-255b-481d-88bc-946ed1b7e057\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") "
Sep 09 04:20:13.000915 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.000895    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-home\") pod \"559c5847-255b-481d-88bc-946ed1b7e057\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") "
Sep 09 04:20:13.000966 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.000911    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "559c5847-255b-481d-88bc-946ed1b7e057" (UID: "559c5847-255b-481d-88bc-946ed1b7e057"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:20:13.000966 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.000948    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-dshm\") pod \"559c5847-255b-481d-88bc-946ed1b7e057\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") "
Sep 09 04:20:13.001125 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.000993    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-model-cache\") pod \"559c5847-255b-481d-88bc-946ed1b7e057\" (UID: \"559c5847-255b-481d-88bc-946ed1b7e057\") "
Sep 09 04:20:13.001178 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.001155    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-home" (OuterVolumeSpecName: "home") pod "559c5847-255b-481d-88bc-946ed1b7e057" (UID: "559c5847-255b-481d-88bc-946ed1b7e057"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:20:13.001257 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.001232    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-model-cache" (OuterVolumeSpecName: "model-cache") pod "559c5847-255b-481d-88bc-946ed1b7e057" (UID: "559c5847-255b-481d-88bc-946ed1b7e057"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:20:13.001571 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.001541    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:20:13.001571 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.001565    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:20:13.001849 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.001579    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:20:13.001849 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.001621    2553 scope.go:117] "RemoveContainer" containerID="af1a307c76fb3cff9b9ebcacbc31bc8f6dd6a1f2c1873b359b1dc34761d1902e"
Sep 09 04:20:13.001969 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:20:13.001929    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1a307c76fb3cff9b9ebcacbc31bc8f6dd6a1f2c1873b359b1dc34761d1902e\": container with ID starting with af1a307c76fb3cff9b9ebcacbc31bc8f6dd6a1f2c1873b359b1dc34761d1902e not found: ID does not exist" containerID="af1a307c76fb3cff9b9ebcacbc31bc8f6dd6a1f2c1873b359b1dc34761d1902e"
Sep 09 04:20:13.002030 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.001964    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1a307c76fb3cff9b9ebcacbc31bc8f6dd6a1f2c1873b359b1dc34761d1902e"} err="failed to get container status \"af1a307c76fb3cff9b9ebcacbc31bc8f6dd6a1f2c1873b359b1dc34761d1902e\": rpc error: code = NotFound desc = could not find container \"af1a307c76fb3cff9b9ebcacbc31bc8f6dd6a1f2c1873b359b1dc34761d1902e\": container with ID starting with af1a307c76fb3cff9b9ebcacbc31bc8f6dd6a1f2c1873b359b1dc34761d1902e not found: ID does not exist"
Sep 09 04:20:13.003309 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.003290    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-dshm" (OuterVolumeSpecName: "dshm") pod "559c5847-255b-481d-88bc-946ed1b7e057" (UID: "559c5847-255b-481d-88bc-946ed1b7e057"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:20:13.003523 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.003506    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559c5847-255b-481d-88bc-946ed1b7e057-kube-api-access-njfjn" (OuterVolumeSpecName: "kube-api-access-njfjn") pod "559c5847-255b-481d-88bc-946ed1b7e057" (UID: "559c5847-255b-481d-88bc-946ed1b7e057"). InnerVolumeSpecName "kube-api-access-njfjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:20:13.003523 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.003509    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/559c5847-255b-481d-88bc-946ed1b7e057-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "559c5847-255b-481d-88bc-946ed1b7e057" (UID: "559c5847-255b-481d-88bc-946ed1b7e057"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:20:13.102386 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.102361    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559c5847-255b-481d-88bc-946ed1b7e057-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:20:13.102386 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.102385    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njfjn\" (UniqueName: \"kubernetes.io/projected/559c5847-255b-481d-88bc-946ed1b7e057-kube-api-access-njfjn\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:20:13.102559 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.102394    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/559c5847-255b-481d-88bc-946ed1b7e057-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:20:13.316916 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.316891    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"]
Sep 09 04:20:13.320161 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:13.320143    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-429d71ed/e2e-lifecycle-model-429d71ed-kserve-b8c98c759-x8fhz"]
Sep 09 04:20:14.989603 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:14.989573    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="559c5847-255b-481d-88bc-946ed1b7e057" path="/var/lib/kubelet/pods/559c5847-255b-481d-88bc-946ed1b7e057/volumes"
Sep 09 04:20:17.485192 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:17.485162    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:17.485638 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:17.485223    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:17.497673 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:17.497645    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:18.032713 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:18.032685    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:20:19.158951 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:19.158917    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:19.770326 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:19.770294    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:20:35.045677 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:35.045645    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:35.284632 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:35.284597    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:41.271615 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:41.271585    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:20:41.549738 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:41.549656    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:49.167097 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.167012    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"]
Sep 09 04:20:49.167670 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.167651    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="559c5847-255b-481d-88bc-946ed1b7e057" containerName="main"
Sep 09 04:20:49.167670 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.167671    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="559c5847-255b-481d-88bc-946ed1b7e057" containerName="main"
Sep 09 04:20:49.167817 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.167805    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="559c5847-255b-481d-88bc-946ed1b7e057" containerName="main"
Sep 09 04:20:49.172531 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.172510    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.175886 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.175867    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9c44b5\"/\"kube-root-ca.crt\""
Sep 09 04:20:49.176003 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.175867    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9c44b5\"/\"default-dockercfg-jdl5d\""
Sep 09 04:20:49.176003 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.175879    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9c44b5\"/\"auth-test-model-9c44b5-kserve-self-signed-certs\""
Sep 09 04:20:49.176003 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.175887    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9c44b5\"/\"openshift-service-ca.crt\""
Sep 09 04:20:49.181969 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.181918    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"]
Sep 09 04:20:49.202717 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.202696    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:49.227011 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.226987    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704f7ad4-59be-40a8-90de-c869d9260f34-tls-certs\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.227126 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.227019    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-home\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.227126 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.227049    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8929q\" (UniqueName: \"kubernetes.io/projected/704f7ad4-59be-40a8-90de-c869d9260f34-kube-api-access-8929q\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.227203 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.227156    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-tmp-dir\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.227203 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.227192    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-dshm\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.227278 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.227236    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-model-cache\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.328693 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.328664    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-home\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.328887 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.328720    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8929q\" (UniqueName: \"kubernetes.io/projected/704f7ad4-59be-40a8-90de-c869d9260f34-kube-api-access-8929q\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.328887 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.328787    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-tmp-dir\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.328887 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.328832    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-dshm\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.329068 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.328892    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-model-cache\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.329068 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.328966    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704f7ad4-59be-40a8-90de-c869d9260f34-tls-certs\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.329171 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.329089    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-home\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.329220 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.329181    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-tmp-dir\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.329400 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.329379    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-model-cache\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.332111 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.332064    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-dshm\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.332111 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.332090    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704f7ad4-59be-40a8-90de-c869d9260f34-tls-certs\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.338311 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.338286    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8929q\" (UniqueName: \"kubernetes.io/projected/704f7ad4-59be-40a8-90de-c869d9260f34-kube-api-access-8929q\") pod \"auth-test-model-9c44b5-kserve-79cc44dc5f-252tx\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.486326 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.486303    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:49.759311 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:49.759288    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"]
Sep 09 04:20:49.760894 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:20:49.760862    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod704f7ad4_59be_40a8_90de_c869d9260f34.slice/crio-5ddbd0337a9afdd1e0ac07f4e9ae4f09698d384904d853e719190ad985b47e1b WatchSource:0}: Error finding container 5ddbd0337a9afdd1e0ac07f4e9ae4f09698d384904d853e719190ad985b47e1b: Status 404 returned error can't find the container with id 5ddbd0337a9afdd1e0ac07f4e9ae4f09698d384904d853e719190ad985b47e1b
Sep 09 04:20:50.204236 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:50.204157    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx" event={"ID":"704f7ad4-59be-40a8-90de-c869d9260f34","Type":"ContainerStarted","Data":"b8d47a98b905b6387eaea25078fc7e1498c1abb8edc6e4218098267d329e8f6b"}
Sep 09 04:20:50.204236 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:50.204212    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx" event={"ID":"704f7ad4-59be-40a8-90de-c869d9260f34","Type":"ContainerStarted","Data":"5ddbd0337a9afdd1e0ac07f4e9ae4f09698d384904d853e719190ad985b47e1b"}
Sep 09 04:20:50.227424 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:50.227370    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx" podStartSLOduration=1.227354552 podStartE2EDuration="1.227354552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:20:50.224642352 +0000 UTC m=+1349.751013062" watchObservedRunningTime="2026-09-09 04:20:50.227354552 +0000 UTC m=+1349.753725262"
Sep 09 04:20:57.091454 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:57.091417    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:57.191691 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:57.191658    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:57.319890 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:57.319863    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"]
Sep 09 04:20:57.320093 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:57.320066    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp" podUID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerName="istio-proxy" containerID="cri-o://c7c89193045fb15c8ea33f5d22e4a459e7906b83c58f755d3548e90baccea318" gracePeriod=30
Sep 09 04:20:57.340205 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:57.340178    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:58.585736 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:58.585692    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:58.686638 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:58.686611    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:58.786653 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:58.786624    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:59.381932 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:59.381897    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:59.480788 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:59.480766    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:20:59.486709 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:59.486688    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:59.491460 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:59.488125    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:59.502791 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:59.502772    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:20:59.581654 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:20:59.581626    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:00.271602 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:00.271556    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:21:00.757084 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:00.757053    2553 patch_prober.go:28] interesting pod/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 04:21:00.757236 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:00.757130    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp" podUID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 04:21:00.782499 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:00.782425    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:02.578885 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.578835    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:21:02.761159 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.761136    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-558vx\" (UniqueName: \"kubernetes.io/projected/b7972eed-5e82-4b54-9d41-b36c41787d7d-kube-api-access-558vx\") pod \"b7972eed-5e82-4b54-9d41-b36c41787d7d\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") "
Sep 09 04:21:02.761321 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.761171    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-podinfo\") pod \"b7972eed-5e82-4b54-9d41-b36c41787d7d\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") "
Sep 09 04:21:02.761321 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.761215    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-workload-certs\") pod \"b7972eed-5e82-4b54-9d41-b36c41787d7d\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") "
Sep 09 04:21:02.761321 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.761264    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-envoy\") pod \"b7972eed-5e82-4b54-9d41-b36c41787d7d\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") "
Sep 09 04:21:02.761321 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.761296    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-data\") pod \"b7972eed-5e82-4b54-9d41-b36c41787d7d\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") "
Sep 09 04:21:02.761571 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.761343    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-credential-socket\") pod \"b7972eed-5e82-4b54-9d41-b36c41787d7d\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") "
Sep 09 04:21:02.761571 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.761379    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-token\") pod \"b7972eed-5e82-4b54-9d41-b36c41787d7d\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") "
Sep 09 04:21:02.761571 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.761478    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-workload-socket\") pod \"b7972eed-5e82-4b54-9d41-b36c41787d7d\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") "
Sep 09 04:21:02.761571 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.761528    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7972eed-5e82-4b54-9d41-b36c41787d7d-istiod-ca-cert\") pod \"b7972eed-5e82-4b54-9d41-b36c41787d7d\" (UID: \"b7972eed-5e82-4b54-9d41-b36c41787d7d\") "
Sep 09 04:21:02.761777 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.761636    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b7972eed-5e82-4b54-9d41-b36c41787d7d" (UID: "b7972eed-5e82-4b54-9d41-b36c41787d7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:21:02.761777 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.761646    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b7972eed-5e82-4b54-9d41-b36c41787d7d" (UID: "b7972eed-5e82-4b54-9d41-b36c41787d7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:21:02.761777 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.761756    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-data" (OuterVolumeSpecName: "istio-data") pod "b7972eed-5e82-4b54-9d41-b36c41787d7d" (UID: "b7972eed-5e82-4b54-9d41-b36c41787d7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:21:02.761963 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.761936    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b7972eed-5e82-4b54-9d41-b36c41787d7d" (UID: "b7972eed-5e82-4b54-9d41-b36c41787d7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:21:02.762023 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.761985    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7972eed-5e82-4b54-9d41-b36c41787d7d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b7972eed-5e82-4b54-9d41-b36c41787d7d" (UID: "b7972eed-5e82-4b54-9d41-b36c41787d7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:21:02.762023 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.762000    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-workload-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:21:02.762118 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.762022    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-data\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:21:02.762118 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.762035    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-credential-socket\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:21:02.763754 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.763727    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b7972eed-5e82-4b54-9d41-b36c41787d7d" (UID: "b7972eed-5e82-4b54-9d41-b36c41787d7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:21:02.763850 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.763798    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-token" (OuterVolumeSpecName: "istio-token") pod "b7972eed-5e82-4b54-9d41-b36c41787d7d" (UID: "b7972eed-5e82-4b54-9d41-b36c41787d7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:21:02.764046 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.764029    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b7972eed-5e82-4b54-9d41-b36c41787d7d" (UID: "b7972eed-5e82-4b54-9d41-b36c41787d7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:21:02.764113 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.764030    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7972eed-5e82-4b54-9d41-b36c41787d7d-kube-api-access-558vx" (OuterVolumeSpecName: "kube-api-access-558vx") pod "b7972eed-5e82-4b54-9d41-b36c41787d7d" (UID: "b7972eed-5e82-4b54-9d41-b36c41787d7d"). InnerVolumeSpecName "kube-api-access-558vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:21:02.863245 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.863188    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-token\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:21:02.863245 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.863215    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-workload-socket\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:21:02.863245 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.863228    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7972eed-5e82-4b54-9d41-b36c41787d7d-istiod-ca-cert\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:21:02.863245 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.863240    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-558vx\" (UniqueName: \"kubernetes.io/projected/b7972eed-5e82-4b54-9d41-b36c41787d7d-kube-api-access-558vx\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:21:02.863435 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.863252    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-podinfo\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:21:02.863435 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:02.863264    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7972eed-5e82-4b54-9d41-b36c41787d7d-istio-envoy\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:21:03.277743 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:03.277711    2553 generic.go:356] "Generic (PLEG): container finished" podID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerID="c7c89193045fb15c8ea33f5d22e4a459e7906b83c58f755d3548e90baccea318" exitCode=0
Sep 09 04:21:03.277908 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:03.277759    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"
Sep 09 04:21:03.277908 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:03.277791    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp" event={"ID":"b7972eed-5e82-4b54-9d41-b36c41787d7d","Type":"ContainerDied","Data":"c7c89193045fb15c8ea33f5d22e4a459e7906b83c58f755d3548e90baccea318"}
Sep 09 04:21:03.277908 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:03.277846    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp" event={"ID":"b7972eed-5e82-4b54-9d41-b36c41787d7d","Type":"ContainerDied","Data":"992fcd257be5ce3c8a208e9f28d1d622226cd72a859596101418114249099556"}
Sep 09 04:21:03.277908 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:03.277868    2553 scope.go:117] "RemoveContainer" containerID="c7c89193045fb15c8ea33f5d22e4a459e7906b83c58f755d3548e90baccea318"
Sep 09 04:21:03.287358 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:03.287341    2553 scope.go:117] "RemoveContainer" containerID="749df95076e0335bb9fed5bef6a3b9d68a17ee856c18d4c71b1726793f54848c"
Sep 09 04:21:03.296138 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:03.296120    2553 scope.go:117] "RemoveContainer" containerID="c7c89193045fb15c8ea33f5d22e4a459e7906b83c58f755d3548e90baccea318"
Sep 09 04:21:03.296495 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:21:03.296473    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c89193045fb15c8ea33f5d22e4a459e7906b83c58f755d3548e90baccea318\": container with ID starting with c7c89193045fb15c8ea33f5d22e4a459e7906b83c58f755d3548e90baccea318 not found: ID does not exist" containerID="c7c89193045fb15c8ea33f5d22e4a459e7906b83c58f755d3548e90baccea318"
Sep 09 04:21:03.296559 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:03.296507    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c89193045fb15c8ea33f5d22e4a459e7906b83c58f755d3548e90baccea318"} err="failed to get container status \"c7c89193045fb15c8ea33f5d22e4a459e7906b83c58f755d3548e90baccea318\": rpc error: code = NotFound desc = could not find container \"c7c89193045fb15c8ea33f5d22e4a459e7906b83c58f755d3548e90baccea318\": container with ID starting with c7c89193045fb15c8ea33f5d22e4a459e7906b83c58f755d3548e90baccea318 not found: ID does not exist"
Sep 09 04:21:03.296559 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:03.296525    2553 scope.go:117] "RemoveContainer" containerID="749df95076e0335bb9fed5bef6a3b9d68a17ee856c18d4c71b1726793f54848c"
Sep 09 04:21:03.296797 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:21:03.296770    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749df95076e0335bb9fed5bef6a3b9d68a17ee856c18d4c71b1726793f54848c\": container with ID starting with 749df95076e0335bb9fed5bef6a3b9d68a17ee856c18d4c71b1726793f54848c not found: ID does not exist" containerID="749df95076e0335bb9fed5bef6a3b9d68a17ee856c18d4c71b1726793f54848c"
Sep 09 04:21:03.296797 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:03.296789    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749df95076e0335bb9fed5bef6a3b9d68a17ee856c18d4c71b1726793f54848c"} err="failed to get container status \"749df95076e0335bb9fed5bef6a3b9d68a17ee856c18d4c71b1726793f54848c\": rpc error: code = NotFound desc = could not find container \"749df95076e0335bb9fed5bef6a3b9d68a17ee856c18d4c71b1726793f54848c\": container with ID starting with 749df95076e0335bb9fed5bef6a3b9d68a17ee856c18d4c71b1726793f54848c not found: ID does not exist"
Sep 09 04:21:03.298345 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:03.298327    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"]
Sep 09 04:21:03.302118 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:03.302093    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-429d71ed-openshift-default-db6767fb4-qr9rp"]
Sep 09 04:21:04.988985 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:04.988955    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7972eed-5e82-4b54-9d41-b36c41787d7d" path="/var/lib/kubelet/pods/b7972eed-5e82-4b54-9d41-b36c41787d7d/volumes"
Sep 09 04:21:05.388436 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:05.388355    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:06.910016 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:06.909980    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:07.009811 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:07.009781    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:07.590698 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:07.590670    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:08.760919 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:08.760887    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:21:13.995465 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:13.995407    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:14.054007 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:14.053977    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:14.129414 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:14.129385    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:14.230148 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:14.230115    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:21.171840 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:21.171810    2553 scope.go:117] "RemoveContainer" containerID="c1bde97f2ca8720ff6f2044ac67b0ca43b53ffff79ccaa777cdf4957581efcb6"
Sep 09 04:21:24.720480 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:24.720430    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:24.906709 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:24.906680    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:28.473760 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:28.473727    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:31.595843 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:31.595805    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:36.069352 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:36.069300    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:21:45.738546 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:45.738515    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:45.844593 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:45.844563    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:47.072855 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:47.072801    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:52.220688 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:52.220651    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:52.276382 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:52.276350    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:52.619475 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:52.619392    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:52.873437 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:52.873372    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:53.423646 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:53.423612    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:56.107918 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:56.107889    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:56.206721 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:56.206690    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:57.415660 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:57.415633    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:57.514705 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:57.514666    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:57.742481 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:57.742416    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:57.928489 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:57.928424    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:58.030131 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:58.030057    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:21:59.487942 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:21:59.487901    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:00.185961 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:00.185924    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:05.071531 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:05.071487    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:22:21.071131 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:21.071052    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:21.199830 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:21.199804    2553 scope.go:117] "RemoveContainer" containerID="ef4a889a3cd0e95ff4976e5111833b9ee334b35d5c292276e25da06b9a6dafb1"
Sep 09 04:22:21.208546 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:21.208526    2553 scope.go:117] "RemoveContainer" containerID="5348aed62fc370d21dda8cbe5b1a29e79e9dc94adba44a3ecd17ec5692706672"
Sep 09 04:22:21.531309 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:21.531277    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:21.836170 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:21.836087    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:22.528742 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:22.528714    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:24.436966 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:24.436926    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:25.291536 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:25.291507    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:25.390885 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:25.390848    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:26.618487 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:26.618431    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:26.718245 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:26.718217    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:26.853896 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:26.853863    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:27.223360 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.223324    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:27.322890 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.322861    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:27.331734 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.331703    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"]
Sep 09 04:22:27.332178 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.332164    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerName="istio-proxy"
Sep 09 04:22:27.332223 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.332185    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerName="istio-proxy"
Sep 09 04:22:27.332223 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.332196    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerName="istio-proxy"
Sep 09 04:22:27.332223 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.332201    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerName="istio-proxy"
Sep 09 04:22:27.332317 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.332297    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerName="istio-proxy"
Sep 09 04:22:27.332317 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.332304    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7972eed-5e82-4b54-9d41-b36c41787d7d" containerName="istio-proxy"
Sep 09 04:22:27.336876 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.336860    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"
Sep 09 04:22:27.338992 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.338975    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-acfe5b\""
Sep 09 04:22:27.339072 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.338979    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-acfe5b-dockercfg-qht8m\""
Sep 09 04:22:27.345786 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.345763    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"]
Sep 09 04:22:27.435126 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.435100    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwgxp\" (UniqueName: \"kubernetes.io/projected/c85781a4-f4f9-4fe8-96ce-3e437d6cc24f-kube-api-access-fwgxp\") pod \"payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb\" (UID: \"c85781a4-f4f9-4fe8-96ce-3e437d6cc24f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"
Sep 09 04:22:27.435293 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.435237    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85781a4-f4f9-4fe8-96ce-3e437d6cc24f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb\" (UID: \"c85781a4-f4f9-4fe8-96ce-3e437d6cc24f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"
Sep 09 04:22:27.536463 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.536355    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85781a4-f4f9-4fe8-96ce-3e437d6cc24f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb\" (UID: \"c85781a4-f4f9-4fe8-96ce-3e437d6cc24f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"
Sep 09 04:22:27.536463 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.536435    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwgxp\" (UniqueName: \"kubernetes.io/projected/c85781a4-f4f9-4fe8-96ce-3e437d6cc24f-kube-api-access-fwgxp\") pod \"payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb\" (UID: \"c85781a4-f4f9-4fe8-96ce-3e437d6cc24f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"
Sep 09 04:22:27.537073 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.537047    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85781a4-f4f9-4fe8-96ce-3e437d6cc24f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb\" (UID: \"c85781a4-f4f9-4fe8-96ce-3e437d6cc24f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"
Sep 09 04:22:27.544889 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.544866    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwgxp\" (UniqueName: \"kubernetes.io/projected/c85781a4-f4f9-4fe8-96ce-3e437d6cc24f-kube-api-access-fwgxp\") pod \"payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb\" (UID: \"c85781a4-f4f9-4fe8-96ce-3e437d6cc24f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"
Sep 09 04:22:27.648748 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.648719    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"
Sep 09 04:22:27.924476 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:27.924432    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"]
Sep 09 04:22:27.926707 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:22:27.926680    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85781a4_f4f9_4fe8_96ce_3e437d6cc24f.slice/crio-d9ad8b64278447c70aa1782132c53fbb38c5e826aa80b273357b4857ee72e555 WatchSource:0}: Error finding container d9ad8b64278447c70aa1782132c53fbb38c5e826aa80b273357b4857ee72e555: Status 404 returned error can't find the container with id d9ad8b64278447c70aa1782132c53fbb38c5e826aa80b273357b4857ee72e555
Sep 09 04:22:28.752290 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:28.752258    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb" event={"ID":"c85781a4-f4f9-4fe8-96ce-3e437d6cc24f","Type":"ContainerStarted","Data":"8885a16afc5cb9d8e19b8bd30fcc61b361a42de12879efddea146dcdb6c9e2ef"}
Sep 09 04:22:28.752672 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:28.752328    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb" event={"ID":"c85781a4-f4f9-4fe8-96ce-3e437d6cc24f","Type":"ContainerStarted","Data":"d9ad8b64278447c70aa1782132c53fbb38c5e826aa80b273357b4857ee72e555"}
Sep 09 04:22:28.752672 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:28.752392    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"
Sep 09 04:22:28.769909 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:28.769867    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb" podStartSLOduration=1.769854037 podStartE2EDuration="1.769854037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:22:28.767204132 +0000 UTC m=+1448.293574842" watchObservedRunningTime="2026-09-09 04:22:28.769854037 +0000 UTC m=+1448.296224747"
Sep 09 04:22:39.760591 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:39.760559    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"
Sep 09 04:22:39.782867 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:39.782841    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:22:51.164793 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:51.164763    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:53.216929 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:53.216896    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n"]
Sep 09 04:22:53.220746 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:53.220730    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n"
Sep 09 04:22:53.223180 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:53.223164    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:22:53.223259 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:53.223205    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:22:53.227770 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:53.227748    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n"]
Sep 09 04:22:53.344132 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:53.344094    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:22:53.372565 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:53.372541    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb950596-3b1f-4a76-965a-0c163630c3b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n\" (UID: \"eb950596-3b1f-4a76-965a-0c163630c3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n"
Sep 09 04:22:53.474103 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:53.474076    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb950596-3b1f-4a76-965a-0c163630c3b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n\" (UID: \"eb950596-3b1f-4a76-965a-0c163630c3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n"
Sep 09 04:22:53.474686 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:53.474667    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb950596-3b1f-4a76-965a-0c163630c3b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n\" (UID: \"eb950596-3b1f-4a76-965a-0c163630c3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n"
Sep 09 04:22:53.534720 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:53.534695    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n"
Sep 09 04:22:53.805560 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:53.804691    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n"]
Sep 09 04:22:53.806467 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:22:53.806400    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb950596_3b1f_4a76_965a_0c163630c3b6.slice/crio-fb55226cac1568b4b8d0720accd23425e00ed85d2bc65561f3c100b6d9994f36 WatchSource:0}: Error finding container fb55226cac1568b4b8d0720accd23425e00ed85d2bc65561f3c100b6d9994f36: Status 404 returned error can't find the container with id fb55226cac1568b4b8d0720accd23425e00ed85d2bc65561f3c100b6d9994f36
Sep 09 04:22:53.890927 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:53.890899    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n" event={"ID":"eb950596-3b1f-4a76-965a-0c163630c3b6","Type":"ContainerStarted","Data":"fb55226cac1568b4b8d0720accd23425e00ed85d2bc65561f3c100b6d9994f36"}
Sep 09 04:22:54.897891 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:54.897857    2553 generic.go:356] "Generic (PLEG): container finished" podID="eb950596-3b1f-4a76-965a-0c163630c3b6" containerID="d98495cadf5b7beb2f0b892f7355d04d084509608fc69fe605d8509656265390" exitCode=0
Sep 09 04:22:54.898238 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:54.897897    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n" event={"ID":"eb950596-3b1f-4a76-965a-0c163630c3b6","Type":"ContainerDied","Data":"d98495cadf5b7beb2f0b892f7355d04d084509608fc69fe605d8509656265390"}
Sep 09 04:22:56.040124 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:56.040096    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n"
Sep 09 04:22:56.193951 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:56.193887    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb950596-3b1f-4a76-965a-0c163630c3b6-maas-api-service-ca\") pod \"eb950596-3b1f-4a76-965a-0c163630c3b6\" (UID: \"eb950596-3b1f-4a76-965a-0c163630c3b6\") "
Sep 09 04:22:56.194213 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:56.194191    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb950596-3b1f-4a76-965a-0c163630c3b6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb950596-3b1f-4a76-965a-0c163630c3b6" (UID: "eb950596-3b1f-4a76-965a-0c163630c3b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:22:56.295505 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:56.295482    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb950596-3b1f-4a76-965a-0c163630c3b6-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:22:56.917558 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:56.917526    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n"
Sep 09 04:22:56.917730 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:56.917559    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n" event={"ID":"eb950596-3b1f-4a76-965a-0c163630c3b6","Type":"ContainerDied","Data":"fb55226cac1568b4b8d0720accd23425e00ed85d2bc65561f3c100b6d9994f36"}
Sep 09 04:22:56.917730 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:22:56.917609    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb55226cac1568b4b8d0720accd23425e00ed85d2bc65561f3c100b6d9994f36"
Sep 09 04:23:08.879103 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:08.879065    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:08.978693 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:08.978661    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:08.989712 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:08.989690    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n"]
Sep 09 04:23:08.989839 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:08.989719    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f95c96-ncm9n"]
Sep 09 04:23:09.052423 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:09.052393    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:09.349874 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:09.349840    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:10.991104 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:10.991072    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb950596-3b1f-4a76-965a-0c163630c3b6" path="/var/lib/kubelet/pods/eb950596-3b1f-4a76-965a-0c163630c3b6/volumes"
Sep 09 04:23:11.067276 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:11.067242    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:23:15.520911 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:15.520870    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:15.621069 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:15.621040    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:15.807770 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:15.807690    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:16.138793 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:16.138705    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h"]
Sep 09 04:23:16.139254 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:16.139236    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb950596-3b1f-4a76-965a-0c163630c3b6" containerName="revoke-keys"
Sep 09 04:23:16.139361 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:16.139257    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb950596-3b1f-4a76-965a-0c163630c3b6" containerName="revoke-keys"
Sep 09 04:23:16.139419 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:16.139364    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb950596-3b1f-4a76-965a-0c163630c3b6" containerName="revoke-keys"
Sep 09 04:23:16.143925 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:16.143904    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h"
Sep 09 04:23:16.146434 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:16.146415    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:23:16.147225 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:16.147207    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:23:16.150209 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:16.150189    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h"]
Sep 09 04:23:16.178905 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:16.178881    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a58985e-6ce9-44e0-94b3-5d1835c7d520-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h\" (UID: \"3a58985e-6ce9-44e0-94b3-5d1835c7d520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h"
Sep 09 04:23:16.279692 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:16.279664    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a58985e-6ce9-44e0-94b3-5d1835c7d520-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h\" (UID: \"3a58985e-6ce9-44e0-94b3-5d1835c7d520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h"
Sep 09 04:23:16.280294 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:16.280276    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a58985e-6ce9-44e0-94b3-5d1835c7d520-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h\" (UID: \"3a58985e-6ce9-44e0-94b3-5d1835c7d520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h"
Sep 09 04:23:16.294814 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:16.294787    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:16.456089 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:16.456027    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h"
Sep 09 04:23:16.729800 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:16.729778    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h"]
Sep 09 04:23:16.731877 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:23:16.731853    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a58985e_6ce9_44e0_94b3_5d1835c7d520.slice/crio-ef5a4f880cc21ea7edec5ce4fb01d6d3e6545f0dfbee003f94cf764a7740420f WatchSource:0}: Error finding container ef5a4f880cc21ea7edec5ce4fb01d6d3e6545f0dfbee003f94cf764a7740420f: Status 404 returned error can't find the container with id ef5a4f880cc21ea7edec5ce4fb01d6d3e6545f0dfbee003f94cf764a7740420f
Sep 09 04:23:17.026464 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:17.026359    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h" event={"ID":"3a58985e-6ce9-44e0-94b3-5d1835c7d520","Type":"ContainerStarted","Data":"a9ca130e43501e1c3540d0d76b4a3c6a0dcb668ce467cd05e447a9355d27d60c"}
Sep 09 04:23:17.026464 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:17.026423    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h" event={"ID":"3a58985e-6ce9-44e0-94b3-5d1835c7d520","Type":"ContainerStarted","Data":"ef5a4f880cc21ea7edec5ce4fb01d6d3e6545f0dfbee003f94cf764a7740420f"}
Sep 09 04:23:17.041663 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:17.041606    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h" podStartSLOduration=1.041589111 podStartE2EDuration="1.041589111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:23:17.041408105 +0000 UTC m=+1496.567778817" watchObservedRunningTime="2026-09-09 04:23:17.041589111 +0000 UTC m=+1496.567959821"
Sep 09 04:23:18.033980 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:18.033942    2553 generic.go:356] "Generic (PLEG): container finished" podID="3a58985e-6ce9-44e0-94b3-5d1835c7d520" containerID="a9ca130e43501e1c3540d0d76b4a3c6a0dcb668ce467cd05e447a9355d27d60c" exitCode=0
Sep 09 04:23:18.034347 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:18.033981    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h" event={"ID":"3a58985e-6ce9-44e0-94b3-5d1835c7d520","Type":"ContainerDied","Data":"a9ca130e43501e1c3540d0d76b4a3c6a0dcb668ce467cd05e447a9355d27d60c"}
Sep 09 04:23:19.185874 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:19.185843    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h"
Sep 09 04:23:19.206573 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:19.206543    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a58985e-6ce9-44e0-94b3-5d1835c7d520-maas-api-service-ca\") pod \"3a58985e-6ce9-44e0-94b3-5d1835c7d520\" (UID: \"3a58985e-6ce9-44e0-94b3-5d1835c7d520\") "
Sep 09 04:23:19.206851 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:19.206830    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a58985e-6ce9-44e0-94b3-5d1835c7d520-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a58985e-6ce9-44e0-94b3-5d1835c7d520" (UID: "3a58985e-6ce9-44e0-94b3-5d1835c7d520"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:23:19.307485 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:19.307398    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a58985e-6ce9-44e0-94b3-5d1835c7d520-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:23:20.046904 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:20.046880    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h"
Sep 09 04:23:20.047096 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:20.046920    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h" event={"ID":"3a58985e-6ce9-44e0-94b3-5d1835c7d520","Type":"ContainerDied","Data":"ef5a4f880cc21ea7edec5ce4fb01d6d3e6545f0dfbee003f94cf764a7740420f"}
Sep 09 04:23:20.047096 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:20.046966    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef5a4f880cc21ea7edec5ce4fb01d6d3e6545f0dfbee003f94cf764a7740420f"
Sep 09 04:23:31.672091 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:31.671993    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:31.717790 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:31.717754    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"]
Sep 09 04:23:31.718019 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:31.717995    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb" podUID="c85781a4-f4f9-4fe8-96ce-3e437d6cc24f" containerName="payload-pre-processing" containerID="cri-o://8885a16afc5cb9d8e19b8bd30fcc61b361a42de12879efddea146dcdb6c9e2ef" gracePeriod=30
Sep 09 04:23:31.879879 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:31.879819    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h"]
Sep 09 04:23:31.882919 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:31.882897    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-acfe5b-bzv6h"]
Sep 09 04:23:31.961686 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:31.961664    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"
Sep 09 04:23:32.026487 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.026438    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwgxp\" (UniqueName: \"kubernetes.io/projected/c85781a4-f4f9-4fe8-96ce-3e437d6cc24f-kube-api-access-fwgxp\") pod \"c85781a4-f4f9-4fe8-96ce-3e437d6cc24f\" (UID: \"c85781a4-f4f9-4fe8-96ce-3e437d6cc24f\") "
Sep 09 04:23:32.026600 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.026592    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85781a4-f4f9-4fe8-96ce-3e437d6cc24f-plugins-config-volume\") pod \"c85781a4-f4f9-4fe8-96ce-3e437d6cc24f\" (UID: \"c85781a4-f4f9-4fe8-96ce-3e437d6cc24f\") "
Sep 09 04:23:32.026922 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.026897    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85781a4-f4f9-4fe8-96ce-3e437d6cc24f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c85781a4-f4f9-4fe8-96ce-3e437d6cc24f" (UID: "c85781a4-f4f9-4fe8-96ce-3e437d6cc24f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:23:32.028437 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.028413    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85781a4-f4f9-4fe8-96ce-3e437d6cc24f-kube-api-access-fwgxp" (OuterVolumeSpecName: "kube-api-access-fwgxp") pod "c85781a4-f4f9-4fe8-96ce-3e437d6cc24f" (UID: "c85781a4-f4f9-4fe8-96ce-3e437d6cc24f"). InnerVolumeSpecName "kube-api-access-fwgxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:23:32.114269 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.114237    2553 generic.go:356] "Generic (PLEG): container finished" podID="c85781a4-f4f9-4fe8-96ce-3e437d6cc24f" containerID="8885a16afc5cb9d8e19b8bd30fcc61b361a42de12879efddea146dcdb6c9e2ef" exitCode=0
Sep 09 04:23:32.114423 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.114282    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"
Sep 09 04:23:32.114423 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.114321    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb" event={"ID":"c85781a4-f4f9-4fe8-96ce-3e437d6cc24f","Type":"ContainerDied","Data":"8885a16afc5cb9d8e19b8bd30fcc61b361a42de12879efddea146dcdb6c9e2ef"}
Sep 09 04:23:32.114423 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.114373    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb" event={"ID":"c85781a4-f4f9-4fe8-96ce-3e437d6cc24f","Type":"ContainerDied","Data":"d9ad8b64278447c70aa1782132c53fbb38c5e826aa80b273357b4857ee72e555"}
Sep 09 04:23:32.114423 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.114405    2553 scope.go:117] "RemoveContainer" containerID="8885a16afc5cb9d8e19b8bd30fcc61b361a42de12879efddea146dcdb6c9e2ef"
Sep 09 04:23:32.126000 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.125980    2553 scope.go:117] "RemoveContainer" containerID="8885a16afc5cb9d8e19b8bd30fcc61b361a42de12879efddea146dcdb6c9e2ef"
Sep 09 04:23:32.126419 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:23:32.126384    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8885a16afc5cb9d8e19b8bd30fcc61b361a42de12879efddea146dcdb6c9e2ef\": container with ID starting with 8885a16afc5cb9d8e19b8bd30fcc61b361a42de12879efddea146dcdb6c9e2ef not found: ID does not exist" containerID="8885a16afc5cb9d8e19b8bd30fcc61b361a42de12879efddea146dcdb6c9e2ef"
Sep 09 04:23:32.126544 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.126433    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8885a16afc5cb9d8e19b8bd30fcc61b361a42de12879efddea146dcdb6c9e2ef"} err="failed to get container status \"8885a16afc5cb9d8e19b8bd30fcc61b361a42de12879efddea146dcdb6c9e2ef\": rpc error: code = NotFound desc = could not find container \"8885a16afc5cb9d8e19b8bd30fcc61b361a42de12879efddea146dcdb6c9e2ef\": container with ID starting with 8885a16afc5cb9d8e19b8bd30fcc61b361a42de12879efddea146dcdb6c9e2ef not found: ID does not exist"
Sep 09 04:23:32.128095 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.128077    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85781a4-f4f9-4fe8-96ce-3e437d6cc24f-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:23:32.128161 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.128098    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwgxp\" (UniqueName: \"kubernetes.io/projected/c85781a4-f4f9-4fe8-96ce-3e437d6cc24f-kube-api-access-fwgxp\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:23:32.136775 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.136755    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"]
Sep 09 04:23:32.141036 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.141015    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-acfe5b-5b7d566786-d5ckb"]
Sep 09 04:23:32.989695 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.989663    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a58985e-6ce9-44e0-94b3-5d1835c7d520" path="/var/lib/kubelet/pods/3a58985e-6ce9-44e0-94b3-5d1835c7d520/volumes"
Sep 09 04:23:32.990072 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:32.990012    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c85781a4-f4f9-4fe8-96ce-3e437d6cc24f" path="/var/lib/kubelet/pods/c85781a4-f4f9-4fe8-96ce-3e437d6cc24f/volumes"
Sep 09 04:23:33.688253 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:33.688206    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:38.161474 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:38.161427    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:23:39.047886 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.047856    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:39.113408 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.113379    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:39.400919 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.400831    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:39.661604 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.661522    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d"]
Sep 09 04:23:39.661963 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.661948    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a58985e-6ce9-44e0-94b3-5d1835c7d520" containerName="revoke-keys"
Sep 09 04:23:39.661963 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.661962    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a58985e-6ce9-44e0-94b3-5d1835c7d520" containerName="revoke-keys"
Sep 09 04:23:39.662120 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.661973    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85781a4-f4f9-4fe8-96ce-3e437d6cc24f" containerName="payload-pre-processing"
Sep 09 04:23:39.662120 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.661979    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85781a4-f4f9-4fe8-96ce-3e437d6cc24f" containerName="payload-pre-processing"
Sep 09 04:23:39.662120 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.662055    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c85781a4-f4f9-4fe8-96ce-3e437d6cc24f" containerName="payload-pre-processing"
Sep 09 04:23:39.662120 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.662065    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a58985e-6ce9-44e0-94b3-5d1835c7d520" containerName="revoke-keys"
Sep 09 04:23:39.666942 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.666925    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d"
Sep 09 04:23:39.672256 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.670137    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:23:39.672256 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.670268    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:23:39.676459 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.675945    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d"]
Sep 09 04:23:39.790767 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.790739    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:39.798900 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.798866    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3c8b2d-3b28-453d-90f6-684e91439d9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d\" (UID: \"ab3c8b2d-3b28-453d-90f6-684e91439d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d"
Sep 09 04:23:39.899978 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.899946    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3c8b2d-3b28-453d-90f6-684e91439d9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d\" (UID: \"ab3c8b2d-3b28-453d-90f6-684e91439d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d"
Sep 09 04:23:39.900553 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.900533    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3c8b2d-3b28-453d-90f6-684e91439d9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d\" (UID: \"ab3c8b2d-3b28-453d-90f6-684e91439d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d"
Sep 09 04:23:39.988939 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:39.988918    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d"
Sep 09 04:23:40.258848 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:40.258820    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d"]
Sep 09 04:23:40.260283 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:23:40.260242    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3c8b2d_3b28_453d_90f6_684e91439d9c.slice/crio-a9000b947b24e705b6d2545252f0a41de2d5d919a70682c4373e0035369123f8 WatchSource:0}: Error finding container a9000b947b24e705b6d2545252f0a41de2d5d919a70682c4373e0035369123f8: Status 404 returned error can't find the container with id a9000b947b24e705b6d2545252f0a41de2d5d919a70682c4373e0035369123f8
Sep 09 04:23:41.165135 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:41.165051    2553 generic.go:356] "Generic (PLEG): container finished" podID="ab3c8b2d-3b28-453d-90f6-684e91439d9c" containerID="db536a41553cac559ace08e6f7a85e206cb98c287e255bfb3d1b5aec20acf127" exitCode=0
Sep 09 04:23:41.165525 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:41.165153    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d" event={"ID":"ab3c8b2d-3b28-453d-90f6-684e91439d9c","Type":"ContainerDied","Data":"db536a41553cac559ace08e6f7a85e206cb98c287e255bfb3d1b5aec20acf127"}
Sep 09 04:23:41.165525 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:41.165188    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d" event={"ID":"ab3c8b2d-3b28-453d-90f6-684e91439d9c","Type":"ContainerStarted","Data":"a9000b947b24e705b6d2545252f0a41de2d5d919a70682c4373e0035369123f8"}
Sep 09 04:23:42.307089 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:42.307069    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d"
Sep 09 04:23:42.424282 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:42.424176    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3c8b2d-3b28-453d-90f6-684e91439d9c-maas-api-service-ca\") pod \"ab3c8b2d-3b28-453d-90f6-684e91439d9c\" (UID: \"ab3c8b2d-3b28-453d-90f6-684e91439d9c\") "
Sep 09 04:23:42.424549 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:42.424526    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3c8b2d-3b28-453d-90f6-684e91439d9c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ab3c8b2d-3b28-453d-90f6-684e91439d9c" (UID: "ab3c8b2d-3b28-453d-90f6-684e91439d9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:23:42.525319 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:42.525283    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3c8b2d-3b28-453d-90f6-684e91439d9c-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:23:43.177482 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:43.177434    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d"
Sep 09 04:23:43.177482 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:43.177471    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d" event={"ID":"ab3c8b2d-3b28-453d-90f6-684e91439d9c","Type":"ContainerDied","Data":"a9000b947b24e705b6d2545252f0a41de2d5d919a70682c4373e0035369123f8"}
Sep 09 04:23:43.177665 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:43.177513    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9000b947b24e705b6d2545252f0a41de2d5d919a70682c4373e0035369123f8"
Sep 09 04:23:44.057212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:44.057172    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:48.797816 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:48.797779    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:48.899586 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:48.899560    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:49.918736 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:49.918696    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:50.541943 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:50.541913    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:50.591356 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:50.591323    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"]
Sep 09 04:23:50.591633 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:50.591607    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb" podUID="5e4bda94-fbc9-476f-8346-6f19a253b852" containerName="payload-pre-processing" containerID="cri-o://a4528e4c61ad0d35c3c781ec3e6856e3db5a634e42874347e77876260e8ba897" gracePeriod=30
Sep 09 04:23:50.620275 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:50.620252    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:50.691201 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:50.691179    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:50.789090 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:50.789060    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:50.795150 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:50.795074    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d"]
Sep 09 04:23:50.800987 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:50.800958    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-46ddac-mcr4d"]
Sep 09 04:23:50.848961 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:50.848944    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"
Sep 09 04:23:50.873924 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:50.873893    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:50.989702 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:50.989663    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab3c8b2d-3b28-453d-90f6-684e91439d9c" path="/var/lib/kubelet/pods/ab3c8b2d-3b28-453d-90f6-684e91439d9c/volumes"
Sep 09 04:23:51.002953 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.002932    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrdzz\" (UniqueName: \"kubernetes.io/projected/5e4bda94-fbc9-476f-8346-6f19a253b852-kube-api-access-nrdzz\") pod \"5e4bda94-fbc9-476f-8346-6f19a253b852\" (UID: \"5e4bda94-fbc9-476f-8346-6f19a253b852\") "
Sep 09 04:23:51.003115 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.003031    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4bda94-fbc9-476f-8346-6f19a253b852-plugins-config-volume\") pod \"5e4bda94-fbc9-476f-8346-6f19a253b852\" (UID: \"5e4bda94-fbc9-476f-8346-6f19a253b852\") "
Sep 09 04:23:51.003369 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.003349    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4bda94-fbc9-476f-8346-6f19a253b852-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e4bda94-fbc9-476f-8346-6f19a253b852" (UID: "5e4bda94-fbc9-476f-8346-6f19a253b852"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:23:51.005154 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.005124    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4bda94-fbc9-476f-8346-6f19a253b852-kube-api-access-nrdzz" (OuterVolumeSpecName: "kube-api-access-nrdzz") pod "5e4bda94-fbc9-476f-8346-6f19a253b852" (UID: "5e4bda94-fbc9-476f-8346-6f19a253b852"). InnerVolumeSpecName "kube-api-access-nrdzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:23:51.104001 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.103948    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrdzz\" (UniqueName: \"kubernetes.io/projected/5e4bda94-fbc9-476f-8346-6f19a253b852-kube-api-access-nrdzz\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:23:51.104001 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.103970    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4bda94-fbc9-476f-8346-6f19a253b852-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:23:51.151481 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.151437    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:51.225267 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.225242    2553 generic.go:356] "Generic (PLEG): container finished" podID="5e4bda94-fbc9-476f-8346-6f19a253b852" containerID="a4528e4c61ad0d35c3c781ec3e6856e3db5a634e42874347e77876260e8ba897" exitCode=0
Sep 09 04:23:51.225416 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.225294    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"
Sep 09 04:23:51.225416 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.225344    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb" event={"ID":"5e4bda94-fbc9-476f-8346-6f19a253b852","Type":"ContainerDied","Data":"a4528e4c61ad0d35c3c781ec3e6856e3db5a634e42874347e77876260e8ba897"}
Sep 09 04:23:51.225416 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.225396    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb" event={"ID":"5e4bda94-fbc9-476f-8346-6f19a253b852","Type":"ContainerDied","Data":"7b7175349b3b6e4279608fbc66b71484ad0a50ac9b407d5a91b6f6c884e240aa"}
Sep 09 04:23:51.225416 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.225417    2553 scope.go:117] "RemoveContainer" containerID="a4528e4c61ad0d35c3c781ec3e6856e3db5a634e42874347e77876260e8ba897"
Sep 09 04:23:51.234887 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.234870    2553 scope.go:117] "RemoveContainer" containerID="a4528e4c61ad0d35c3c781ec3e6856e3db5a634e42874347e77876260e8ba897"
Sep 09 04:23:51.235144 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:23:51.235128    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4528e4c61ad0d35c3c781ec3e6856e3db5a634e42874347e77876260e8ba897\": container with ID starting with a4528e4c61ad0d35c3c781ec3e6856e3db5a634e42874347e77876260e8ba897 not found: ID does not exist" containerID="a4528e4c61ad0d35c3c781ec3e6856e3db5a634e42874347e77876260e8ba897"
Sep 09 04:23:51.235179 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.235152    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4528e4c61ad0d35c3c781ec3e6856e3db5a634e42874347e77876260e8ba897"} err="failed to get container status \"a4528e4c61ad0d35c3c781ec3e6856e3db5a634e42874347e77876260e8ba897\": rpc error: code = NotFound desc = could not find container \"a4528e4c61ad0d35c3c781ec3e6856e3db5a634e42874347e77876260e8ba897\": container with ID starting with a4528e4c61ad0d35c3c781ec3e6856e3db5a634e42874347e77876260e8ba897 not found: ID does not exist"
Sep 09 04:23:51.246689 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.246671    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"]
Sep 09 04:23:51.250137 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:51.250117    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-46ddac-597dcd7c46-xvmgb"]
Sep 09 04:23:52.989407 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:52.989372    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e4bda94-fbc9-476f-8346-6f19a253b852" path="/var/lib/kubelet/pods/5e4bda94-fbc9-476f-8346-6f19a253b852/volumes"
Sep 09 04:23:57.865550 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:57.865512    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:57.923908 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:57.923880    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:58.135120 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.135038    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:58.462708 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.462633    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v"]
Sep 09 04:23:58.463190 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.463175    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e4bda94-fbc9-476f-8346-6f19a253b852" containerName="payload-pre-processing"
Sep 09 04:23:58.463247 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.463193    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e4bda94-fbc9-476f-8346-6f19a253b852" containerName="payload-pre-processing"
Sep 09 04:23:58.463247 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.463220    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab3c8b2d-3b28-453d-90f6-684e91439d9c" containerName="revoke-keys"
Sep 09 04:23:58.463247 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.463228    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab3c8b2d-3b28-453d-90f6-684e91439d9c" containerName="revoke-keys"
Sep 09 04:23:58.463366 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.463338    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab3c8b2d-3b28-453d-90f6-684e91439d9c" containerName="revoke-keys"
Sep 09 04:23:58.463366 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.463352    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e4bda94-fbc9-476f-8346-6f19a253b852" containerName="payload-pre-processing"
Sep 09 04:23:58.468183 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.468162    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v"
Sep 09 04:23:58.470903 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.470878    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:23:58.471039 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.470878    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:23:58.472921 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.472897    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v"]
Sep 09 04:23:58.569717 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.569685    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/162a095d-833a-45fb-adc1-a6b77a44a1ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v\" (UID: \"162a095d-833a-45fb-adc1-a6b77a44a1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v"
Sep 09 04:23:58.597676 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.597647    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:23:58.670720 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.670692    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/162a095d-833a-45fb-adc1-a6b77a44a1ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v\" (UID: \"162a095d-833a-45fb-adc1-a6b77a44a1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v"
Sep 09 04:23:58.671264 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.671245    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/162a095d-833a-45fb-adc1-a6b77a44a1ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v\" (UID: \"162a095d-833a-45fb-adc1-a6b77a44a1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v"
Sep 09 04:23:58.780233 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:58.780214    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v"
Sep 09 04:23:59.052703 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:59.052612    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v"]
Sep 09 04:23:59.055879 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:23:59.055852    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162a095d_833a_45fb_adc1_a6b77a44a1ca.slice/crio-72a24060cf40e1b181a4ba1121215852e780b8aadbd6f24a4155dad60066ec96 WatchSource:0}: Error finding container 72a24060cf40e1b181a4ba1121215852e780b8aadbd6f24a4155dad60066ec96: Status 404 returned error can't find the container with id 72a24060cf40e1b181a4ba1121215852e780b8aadbd6f24a4155dad60066ec96
Sep 09 04:23:59.273715 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:59.273687    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v" event={"ID":"162a095d-833a-45fb-adc1-a6b77a44a1ca","Type":"ContainerStarted","Data":"c8e4d2327a27622b9258963ebd379a68fb5fe0be1372e2f80dadbbd5428e15eb"}
Sep 09 04:23:59.273823 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:59.273741    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v" event={"ID":"162a095d-833a-45fb-adc1-a6b77a44a1ca","Type":"ContainerStarted","Data":"72a24060cf40e1b181a4ba1121215852e780b8aadbd6f24a4155dad60066ec96"}
Sep 09 04:23:59.292864 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:23:59.292823    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v" podStartSLOduration=1.292810488 podStartE2EDuration="1.292810488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:23:59.29059209 +0000 UTC m=+1538.816962798" watchObservedRunningTime="2026-09-09 04:23:59.292810488 +0000 UTC m=+1538.819181196"
Sep 09 04:24:00.280428 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:00.280398    2553 generic.go:356] "Generic (PLEG): container finished" podID="162a095d-833a-45fb-adc1-a6b77a44a1ca" containerID="c8e4d2327a27622b9258963ebd379a68fb5fe0be1372e2f80dadbbd5428e15eb" exitCode=0
Sep 09 04:24:00.280828 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:00.280478    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v" event={"ID":"162a095d-833a-45fb-adc1-a6b77a44a1ca","Type":"ContainerDied","Data":"c8e4d2327a27622b9258963ebd379a68fb5fe0be1372e2f80dadbbd5428e15eb"}
Sep 09 04:24:01.421612 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:01.421590    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v"
Sep 09 04:24:01.497510 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:01.497484    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/162a095d-833a-45fb-adc1-a6b77a44a1ca-maas-api-service-ca\") pod \"162a095d-833a-45fb-adc1-a6b77a44a1ca\" (UID: \"162a095d-833a-45fb-adc1-a6b77a44a1ca\") "
Sep 09 04:24:01.497825 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:01.497804    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162a095d-833a-45fb-adc1-a6b77a44a1ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "162a095d-833a-45fb-adc1-a6b77a44a1ca" (UID: "162a095d-833a-45fb-adc1-a6b77a44a1ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:24:01.598661 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:01.598598    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/162a095d-833a-45fb-adc1-a6b77a44a1ca-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:01.652539 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:01.652511    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:02.293274 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:02.293246    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v"
Sep 09 04:24:02.293274 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:02.293260    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v" event={"ID":"162a095d-833a-45fb-adc1-a6b77a44a1ca","Type":"ContainerDied","Data":"72a24060cf40e1b181a4ba1121215852e780b8aadbd6f24a4155dad60066ec96"}
Sep 09 04:24:02.293555 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:02.293307    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72a24060cf40e1b181a4ba1121215852e780b8aadbd6f24a4155dad60066ec96"
Sep 09 04:24:06.659168 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:06.659140    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"]
Sep 09 04:24:06.668350 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:06.668325    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"]
Sep 09 04:24:06.680533 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:06.680512    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"]
Sep 09 04:24:06.749613 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:06.749577    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:08.956185 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:08.956155    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:09.006178 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.006144    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"]
Sep 09 04:24:09.006460 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.006374    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59" podUID="393d7150-22f6-48a0-a718-c590930bea06" containerName="payload-processing" containerID="cri-o://ca6513ad4b8ed1a53ba4ec5fe470f8b2736dcf1d5e76ed8fab0154d8f543841b" gracePeriod=30
Sep 09 04:24:09.021055 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.021009    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:09.080347 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.080313    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:09.162917 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.162882    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v"]
Sep 09 04:24:09.163766 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.163739    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:09.166149 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.166128    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f2be5-klc2v"]
Sep 09 04:24:09.223767 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.223741    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:09.271131 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.271104    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"
Sep 09 04:24:09.332015 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.331987    2553 generic.go:356] "Generic (PLEG): container finished" podID="393d7150-22f6-48a0-a718-c590930bea06" containerID="ca6513ad4b8ed1a53ba4ec5fe470f8b2736dcf1d5e76ed8fab0154d8f543841b" exitCode=0
Sep 09 04:24:09.332148 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.332028    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"
Sep 09 04:24:09.332148 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.332076    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59" event={"ID":"393d7150-22f6-48a0-a718-c590930bea06","Type":"ContainerDied","Data":"ca6513ad4b8ed1a53ba4ec5fe470f8b2736dcf1d5e76ed8fab0154d8f543841b"}
Sep 09 04:24:09.332148 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.332123    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59" event={"ID":"393d7150-22f6-48a0-a718-c590930bea06","Type":"ContainerDied","Data":"55fc57c78d324552b4bdcd892a10093252e478dfa5bc80e584bd83c49e996eaa"}
Sep 09 04:24:09.332148 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.332147    2553 scope.go:117] "RemoveContainer" containerID="ca6513ad4b8ed1a53ba4ec5fe470f8b2736dcf1d5e76ed8fab0154d8f543841b"
Sep 09 04:24:09.341317 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.341303    2553 scope.go:117] "RemoveContainer" containerID="ca6513ad4b8ed1a53ba4ec5fe470f8b2736dcf1d5e76ed8fab0154d8f543841b"
Sep 09 04:24:09.341628 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:24:09.341609    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca6513ad4b8ed1a53ba4ec5fe470f8b2736dcf1d5e76ed8fab0154d8f543841b\": container with ID starting with ca6513ad4b8ed1a53ba4ec5fe470f8b2736dcf1d5e76ed8fab0154d8f543841b not found: ID does not exist" containerID="ca6513ad4b8ed1a53ba4ec5fe470f8b2736dcf1d5e76ed8fab0154d8f543841b"
Sep 09 04:24:09.341698 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.341634    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca6513ad4b8ed1a53ba4ec5fe470f8b2736dcf1d5e76ed8fab0154d8f543841b"} err="failed to get container status \"ca6513ad4b8ed1a53ba4ec5fe470f8b2736dcf1d5e76ed8fab0154d8f543841b\": rpc error: code = NotFound desc = could not find container \"ca6513ad4b8ed1a53ba4ec5fe470f8b2736dcf1d5e76ed8fab0154d8f543841b\": container with ID starting with ca6513ad4b8ed1a53ba4ec5fe470f8b2736dcf1d5e76ed8fab0154d8f543841b not found: ID does not exist"
Sep 09 04:24:09.371095 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.371076    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393d7150-22f6-48a0-a718-c590930bea06-plugins-config-volume\") pod \"393d7150-22f6-48a0-a718-c590930bea06\" (UID: \"393d7150-22f6-48a0-a718-c590930bea06\") "
Sep 09 04:24:09.371172 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.371118    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26c7n\" (UniqueName: \"kubernetes.io/projected/393d7150-22f6-48a0-a718-c590930bea06-kube-api-access-26c7n\") pod \"393d7150-22f6-48a0-a718-c590930bea06\" (UID: \"393d7150-22f6-48a0-a718-c590930bea06\") "
Sep 09 04:24:09.371403 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.371385    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393d7150-22f6-48a0-a718-c590930bea06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "393d7150-22f6-48a0-a718-c590930bea06" (UID: "393d7150-22f6-48a0-a718-c590930bea06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:24:09.373227 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.373208    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393d7150-22f6-48a0-a718-c590930bea06-kube-api-access-26c7n" (OuterVolumeSpecName: "kube-api-access-26c7n") pod "393d7150-22f6-48a0-a718-c590930bea06" (UID: "393d7150-22f6-48a0-a718-c590930bea06"). InnerVolumeSpecName "kube-api-access-26c7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:24:09.472275 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.472251    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393d7150-22f6-48a0-a718-c590930bea06-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:09.472275 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.472273    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26c7n\" (UniqueName: \"kubernetes.io/projected/393d7150-22f6-48a0-a718-c590930bea06-kube-api-access-26c7n\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:09.520760 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.520677    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:09.655532 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.655510    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"]
Sep 09 04:24:09.657695 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.657676    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9f2be5-5d6df85489-vqs59"]
Sep 09 04:24:09.909169 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:09.909114    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:24:10.991370 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:10.991339    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="162a095d-833a-45fb-adc1-a6b77a44a1ca" path="/var/lib/kubelet/pods/162a095d-833a-45fb-adc1-a6b77a44a1ca/volumes"
Sep 09 04:24:10.991759 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:10.991718    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="393d7150-22f6-48a0-a718-c590930bea06" path="/var/lib/kubelet/pods/393d7150-22f6-48a0-a718-c590930bea06/volumes"
Sep 09 04:24:11.292903 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:11.292792    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:11.393916 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:11.393885    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:12.691966 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:12.691931    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:14.891464 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:14.891409    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:15.868653 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:15.868619    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:15.933271 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:15.933243    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:16.162792 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:16.162707    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:16.770520 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:16.770490    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"]
Sep 09 04:24:16.783591 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:16.783564    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:18.352017 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:18.351991    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"]
Sep 09 04:24:18.369837 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:18.369809    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:21.240209 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:21.240179    2553 scope.go:117] "RemoveContainer" containerID="4ef940435052504137c11ab05d2f02a310bc9bfb8edcc5c915439bbaa94f8d5b"
Sep 09 04:24:21.249200 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:21.249183    2553 scope.go:117] "RemoveContainer" containerID="05ef9c25193f22a82cf7463ba7d595387bfea5e302997f9946036cbcadb5446f"
Sep 09 04:24:21.681595 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:21.681505    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw" podUID="197d60c8-9bf1-4af7-9739-857ae3d05438" containerName="main" containerID="cri-o://540b9b2b20d8a56078a7ce7266171d785fa0b09b27dfb013fe6eb2cc5cc899da" gracePeriod=45
Sep 09 04:24:21.688868 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:21.688832    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p" podUID="55596728-74e8-4c54-ba37-f930bf20e6d0" containerName="main" containerID="cri-o://9fec177b66a229490df6616767b73ca58e96f949c976a5ce52db00100772911a" gracePeriod=45
Sep 09 04:24:21.700189 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:21.700142    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d" podUID="b02176de-51f1-4f4a-8b93-e4008228f719" containerName="main" containerID="cri-o://8999c189c6961769f6bcb144177cb73156f9e092a0d21ada753ea18a294ab32d" gracePeriod=45
Sep 09 04:24:21.986135 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:21.986112    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:24:21.989920 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:21.989900    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:24:21.996361 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:21.996341    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:24:22.082700 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.082669    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55596728-74e8-4c54-ba37-f930bf20e6d0-tls-certs\") pod \"55596728-74e8-4c54-ba37-f930bf20e6d0\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") "
Sep 09 04:24:22.082872 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.082720    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-dshm\") pod \"55596728-74e8-4c54-ba37-f930bf20e6d0\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") "
Sep 09 04:24:22.082872 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.082756    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-dshm\") pod \"197d60c8-9bf1-4af7-9739-857ae3d05438\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") "
Sep 09 04:24:22.082872 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.082783    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02176de-51f1-4f4a-8b93-e4008228f719-tls-certs\") pod \"b02176de-51f1-4f4a-8b93-e4008228f719\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") "
Sep 09 04:24:22.082872 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.082806    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-dshm\") pod \"b02176de-51f1-4f4a-8b93-e4008228f719\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") "
Sep 09 04:24:22.082872 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.082849    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-tmp-dir\") pod \"55596728-74e8-4c54-ba37-f930bf20e6d0\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") "
Sep 09 04:24:22.083125 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.082873    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-home\") pod \"55596728-74e8-4c54-ba37-f930bf20e6d0\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") "
Sep 09 04:24:22.083125 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.082902    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-tmp-dir\") pod \"b02176de-51f1-4f4a-8b93-e4008228f719\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") "
Sep 09 04:24:22.083125 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.082925    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-home\") pod \"197d60c8-9bf1-4af7-9739-857ae3d05438\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") "
Sep 09 04:24:22.083125 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.082952    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-model-cache\") pod \"55596728-74e8-4c54-ba37-f930bf20e6d0\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") "
Sep 09 04:24:22.083125 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.082975    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-model-cache\") pod \"197d60c8-9bf1-4af7-9739-857ae3d05438\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") "
Sep 09 04:24:22.083125 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.083004    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-model-cache\") pod \"b02176de-51f1-4f4a-8b93-e4008228f719\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") "
Sep 09 04:24:22.083125 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.083042    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-tmp-dir\") pod \"197d60c8-9bf1-4af7-9739-857ae3d05438\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") "
Sep 09 04:24:22.083125 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.083108    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197d60c8-9bf1-4af7-9739-857ae3d05438-tls-certs\") pod \"197d60c8-9bf1-4af7-9739-857ae3d05438\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") "
Sep 09 04:24:22.083557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.083131    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkj62\" (UniqueName: \"kubernetes.io/projected/55596728-74e8-4c54-ba37-f930bf20e6d0-kube-api-access-rkj62\") pod \"55596728-74e8-4c54-ba37-f930bf20e6d0\" (UID: \"55596728-74e8-4c54-ba37-f930bf20e6d0\") "
Sep 09 04:24:22.083557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.083161    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgjb8\" (UniqueName: \"kubernetes.io/projected/197d60c8-9bf1-4af7-9739-857ae3d05438-kube-api-access-sgjb8\") pod \"197d60c8-9bf1-4af7-9739-857ae3d05438\" (UID: \"197d60c8-9bf1-4af7-9739-857ae3d05438\") "
Sep 09 04:24:22.083557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.083215    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bblq\" (UniqueName: \"kubernetes.io/projected/b02176de-51f1-4f4a-8b93-e4008228f719-kube-api-access-9bblq\") pod \"b02176de-51f1-4f4a-8b93-e4008228f719\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") "
Sep 09 04:24:22.083557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.083262    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-home\") pod \"b02176de-51f1-4f4a-8b93-e4008228f719\" (UID: \"b02176de-51f1-4f4a-8b93-e4008228f719\") "
Sep 09 04:24:22.083557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.083529    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-home" (OuterVolumeSpecName: "home") pod "55596728-74e8-4c54-ba37-f930bf20e6d0" (UID: "55596728-74e8-4c54-ba37-f930bf20e6d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:22.083790 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.083674    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55596728-74e8-4c54-ba37-f930bf20e6d0" (UID: "55596728-74e8-4c54-ba37-f930bf20e6d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:22.083953 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.083929    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-home" (OuterVolumeSpecName: "home") pod "197d60c8-9bf1-4af7-9739-857ae3d05438" (UID: "197d60c8-9bf1-4af7-9739-857ae3d05438"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:22.084287 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.084036    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b02176de-51f1-4f4a-8b93-e4008228f719" (UID: "b02176de-51f1-4f4a-8b93-e4008228f719"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:22.084287 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.084065    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.084287 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.084082    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.084287 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.084095    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.084605 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.084302    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-home" (OuterVolumeSpecName: "home") pod "b02176de-51f1-4f4a-8b93-e4008228f719" (UID: "b02176de-51f1-4f4a-8b93-e4008228f719"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:22.085629 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.085101    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "55596728-74e8-4c54-ba37-f930bf20e6d0" (UID: "55596728-74e8-4c54-ba37-f930bf20e6d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:22.085629 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.085197    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-model-cache" (OuterVolumeSpecName: "model-cache") pod "b02176de-51f1-4f4a-8b93-e4008228f719" (UID: "b02176de-51f1-4f4a-8b93-e4008228f719"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:22.085629 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.085247    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-model-cache" (OuterVolumeSpecName: "model-cache") pod "197d60c8-9bf1-4af7-9739-857ae3d05438" (UID: "197d60c8-9bf1-4af7-9739-857ae3d05438"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:22.085629 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.085471    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "197d60c8-9bf1-4af7-9739-857ae3d05438" (UID: "197d60c8-9bf1-4af7-9739-857ae3d05438"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:22.086717 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.086684    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-dshm" (OuterVolumeSpecName: "dshm") pod "197d60c8-9bf1-4af7-9739-857ae3d05438" (UID: "197d60c8-9bf1-4af7-9739-857ae3d05438"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:22.087043 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.087020    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-dshm" (OuterVolumeSpecName: "dshm") pod "55596728-74e8-4c54-ba37-f930bf20e6d0" (UID: "55596728-74e8-4c54-ba37-f930bf20e6d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:22.087259 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.087177    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55596728-74e8-4c54-ba37-f930bf20e6d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55596728-74e8-4c54-ba37-f930bf20e6d0" (UID: "55596728-74e8-4c54-ba37-f930bf20e6d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:24:22.087361 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.087337    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02176de-51f1-4f4a-8b93-e4008228f719-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b02176de-51f1-4f4a-8b93-e4008228f719" (UID: "b02176de-51f1-4f4a-8b93-e4008228f719"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:24:22.087499 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.087473    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55596728-74e8-4c54-ba37-f930bf20e6d0-kube-api-access-rkj62" (OuterVolumeSpecName: "kube-api-access-rkj62") pod "55596728-74e8-4c54-ba37-f930bf20e6d0" (UID: "55596728-74e8-4c54-ba37-f930bf20e6d0"). InnerVolumeSpecName "kube-api-access-rkj62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:24:22.087851 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.087821    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-dshm" (OuterVolumeSpecName: "dshm") pod "b02176de-51f1-4f4a-8b93-e4008228f719" (UID: "b02176de-51f1-4f4a-8b93-e4008228f719"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:22.087941 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.087876    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197d60c8-9bf1-4af7-9739-857ae3d05438-kube-api-access-sgjb8" (OuterVolumeSpecName: "kube-api-access-sgjb8") pod "197d60c8-9bf1-4af7-9739-857ae3d05438" (UID: "197d60c8-9bf1-4af7-9739-857ae3d05438"). InnerVolumeSpecName "kube-api-access-sgjb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:24:22.088363 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.088343    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/197d60c8-9bf1-4af7-9739-857ae3d05438-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "197d60c8-9bf1-4af7-9739-857ae3d05438" (UID: "197d60c8-9bf1-4af7-9739-857ae3d05438"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:24:22.088716 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.088696    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02176de-51f1-4f4a-8b93-e4008228f719-kube-api-access-9bblq" (OuterVolumeSpecName: "kube-api-access-9bblq") pod "b02176de-51f1-4f4a-8b93-e4008228f719" (UID: "b02176de-51f1-4f4a-8b93-e4008228f719"). InnerVolumeSpecName "kube-api-access-9bblq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:24:22.185504 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185479    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.185504 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185500    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02176de-51f1-4f4a-8b93-e4008228f719-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.185504 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185509    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.185668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185517    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.185668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185524    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.185668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185531    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.185668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185538    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.185668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185545    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197d60c8-9bf1-4af7-9739-857ae3d05438-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.185668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185552    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197d60c8-9bf1-4af7-9739-857ae3d05438-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.185668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185560    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkj62\" (UniqueName: \"kubernetes.io/projected/55596728-74e8-4c54-ba37-f930bf20e6d0-kube-api-access-rkj62\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.185668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185568    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgjb8\" (UniqueName: \"kubernetes.io/projected/197d60c8-9bf1-4af7-9739-857ae3d05438-kube-api-access-sgjb8\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.185668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185576    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bblq\" (UniqueName: \"kubernetes.io/projected/b02176de-51f1-4f4a-8b93-e4008228f719-kube-api-access-9bblq\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.185668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185584    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02176de-51f1-4f4a-8b93-e4008228f719-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.185668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185592    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55596728-74e8-4c54-ba37-f930bf20e6d0-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.185668 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.185599    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55596728-74e8-4c54-ba37-f930bf20e6d0-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.410510 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.410478    2553 generic.go:356] "Generic (PLEG): container finished" podID="197d60c8-9bf1-4af7-9739-857ae3d05438" containerID="540b9b2b20d8a56078a7ce7266171d785fa0b09b27dfb013fe6eb2cc5cc899da" exitCode=0
Sep 09 04:24:22.410887 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.410529    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"
Sep 09 04:24:22.410887 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.410565    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw" event={"ID":"197d60c8-9bf1-4af7-9739-857ae3d05438","Type":"ContainerDied","Data":"540b9b2b20d8a56078a7ce7266171d785fa0b09b27dfb013fe6eb2cc5cc899da"}
Sep 09 04:24:22.410887 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.410615    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw" event={"ID":"197d60c8-9bf1-4af7-9739-857ae3d05438","Type":"ContainerDied","Data":"73cdb7048f0ef7461f9f2ccf4b86fb612332552e2516e3b4fd0c89151deb859c"}
Sep 09 04:24:22.410887 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.410633    2553 scope.go:117] "RemoveContainer" containerID="540b9b2b20d8a56078a7ce7266171d785fa0b09b27dfb013fe6eb2cc5cc899da"
Sep 09 04:24:22.411989 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.411961    2553 generic.go:356] "Generic (PLEG): container finished" podID="b02176de-51f1-4f4a-8b93-e4008228f719" containerID="8999c189c6961769f6bcb144177cb73156f9e092a0d21ada753ea18a294ab32d" exitCode=0
Sep 09 04:24:22.412041 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.412030    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d" event={"ID":"b02176de-51f1-4f4a-8b93-e4008228f719","Type":"ContainerDied","Data":"8999c189c6961769f6bcb144177cb73156f9e092a0d21ada753ea18a294ab32d"}
Sep 09 04:24:22.412104 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.412036    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"
Sep 09 04:24:22.412104 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.412055    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d" event={"ID":"b02176de-51f1-4f4a-8b93-e4008228f719","Type":"ContainerDied","Data":"80fbb9da524237830bafed3c84e5050e9df43bd82f6b36cd5ef87b4e88a403b6"}
Sep 09 04:24:22.413849 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.413759    2553 generic.go:356] "Generic (PLEG): container finished" podID="55596728-74e8-4c54-ba37-f930bf20e6d0" containerID="9fec177b66a229490df6616767b73ca58e96f949c976a5ce52db00100772911a" exitCode=0
Sep 09 04:24:22.413849 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.413806    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"
Sep 09 04:24:22.413849 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.413805    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p" event={"ID":"55596728-74e8-4c54-ba37-f930bf20e6d0","Type":"ContainerDied","Data":"9fec177b66a229490df6616767b73ca58e96f949c976a5ce52db00100772911a"}
Sep 09 04:24:22.413849 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.413845    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p" event={"ID":"55596728-74e8-4c54-ba37-f930bf20e6d0","Type":"ContainerDied","Data":"8d18c5707332526755bd626267ecb95ee28bebd44f10b66e86d39229333eec7f"}
Sep 09 04:24:22.421834 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.421814    2553 scope.go:117] "RemoveContainer" containerID="540b9b2b20d8a56078a7ce7266171d785fa0b09b27dfb013fe6eb2cc5cc899da"
Sep 09 04:24:22.422126 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:24:22.422105    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"540b9b2b20d8a56078a7ce7266171d785fa0b09b27dfb013fe6eb2cc5cc899da\": container with ID starting with 540b9b2b20d8a56078a7ce7266171d785fa0b09b27dfb013fe6eb2cc5cc899da not found: ID does not exist" containerID="540b9b2b20d8a56078a7ce7266171d785fa0b09b27dfb013fe6eb2cc5cc899da"
Sep 09 04:24:22.422211 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.422132    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"540b9b2b20d8a56078a7ce7266171d785fa0b09b27dfb013fe6eb2cc5cc899da"} err="failed to get container status \"540b9b2b20d8a56078a7ce7266171d785fa0b09b27dfb013fe6eb2cc5cc899da\": rpc error: code = NotFound desc = could not find container \"540b9b2b20d8a56078a7ce7266171d785fa0b09b27dfb013fe6eb2cc5cc899da\": container with ID starting with 540b9b2b20d8a56078a7ce7266171d785fa0b09b27dfb013fe6eb2cc5cc899da not found: ID does not exist"
Sep 09 04:24:22.422211 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.422147    2553 scope.go:117] "RemoveContainer" containerID="8999c189c6961769f6bcb144177cb73156f9e092a0d21ada753ea18a294ab32d"
Sep 09 04:24:22.430429 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.430413    2553 scope.go:117] "RemoveContainer" containerID="8999c189c6961769f6bcb144177cb73156f9e092a0d21ada753ea18a294ab32d"
Sep 09 04:24:22.430697 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:24:22.430681    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8999c189c6961769f6bcb144177cb73156f9e092a0d21ada753ea18a294ab32d\": container with ID starting with 8999c189c6961769f6bcb144177cb73156f9e092a0d21ada753ea18a294ab32d not found: ID does not exist" containerID="8999c189c6961769f6bcb144177cb73156f9e092a0d21ada753ea18a294ab32d"
Sep 09 04:24:22.430752 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.430703    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8999c189c6961769f6bcb144177cb73156f9e092a0d21ada753ea18a294ab32d"} err="failed to get container status \"8999c189c6961769f6bcb144177cb73156f9e092a0d21ada753ea18a294ab32d\": rpc error: code = NotFound desc = could not find container \"8999c189c6961769f6bcb144177cb73156f9e092a0d21ada753ea18a294ab32d\": container with ID starting with 8999c189c6961769f6bcb144177cb73156f9e092a0d21ada753ea18a294ab32d not found: ID does not exist"
Sep 09 04:24:22.430752 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.430717    2553 scope.go:117] "RemoveContainer" containerID="9fec177b66a229490df6616767b73ca58e96f949c976a5ce52db00100772911a"
Sep 09 04:24:22.438547 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.438528    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"]
Sep 09 04:24:22.439056 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.439042    2553 scope.go:117] "RemoveContainer" containerID="9fec177b66a229490df6616767b73ca58e96f949c976a5ce52db00100772911a"
Sep 09 04:24:22.439324 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:24:22.439303    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fec177b66a229490df6616767b73ca58e96f949c976a5ce52db00100772911a\": container with ID starting with 9fec177b66a229490df6616767b73ca58e96f949c976a5ce52db00100772911a not found: ID does not exist" containerID="9fec177b66a229490df6616767b73ca58e96f949c976a5ce52db00100772911a"
Sep 09 04:24:22.439403 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.439328    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fec177b66a229490df6616767b73ca58e96f949c976a5ce52db00100772911a"} err="failed to get container status \"9fec177b66a229490df6616767b73ca58e96f949c976a5ce52db00100772911a\": rpc error: code = NotFound desc = could not find container \"9fec177b66a229490df6616767b73ca58e96f949c976a5ce52db00100772911a\": container with ID starting with 9fec177b66a229490df6616767b73ca58e96f949c976a5ce52db00100772911a not found: ID does not exist"
Sep 09 04:24:22.440762 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.440743    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/premium-w4-85732c-kserve-7f6fc7bb66-wfhbw"]
Sep 09 04:24:22.450384 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.450364    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"]
Sep 09 04:24:22.454501 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.454483    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/unconfigured-w4-85732c-kserve-5c7845f9c8-rtl7d"]
Sep 09 04:24:22.465000 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.464980    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"]
Sep 09 04:24:22.466823 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.466806    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/simulator-w4-85732c-kserve-6f95b54b95-kx42p"]
Sep 09 04:24:22.989737 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.989710    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="197d60c8-9bf1-4af7-9739-857ae3d05438" path="/var/lib/kubelet/pods/197d60c8-9bf1-4af7-9739-857ae3d05438/volumes"
Sep 09 04:24:22.990101 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.990089    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55596728-74e8-4c54-ba37-f930bf20e6d0" path="/var/lib/kubelet/pods/55596728-74e8-4c54-ba37-f930bf20e6d0/volumes"
Sep 09 04:24:22.990491 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:22.990478    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b02176de-51f1-4f4a-8b93-e4008228f719" path="/var/lib/kubelet/pods/b02176de-51f1-4f4a-8b93-e4008228f719/volumes"
Sep 09 04:24:25.890266 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:25.890236    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:31.791117 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:31.791075    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz" podUID="fbaeba7e-5d6f-452b-966a-2e7c1054a17a" containerName="main" containerID="cri-o://3456d3a50f8b32b6ad3a2a7c286fd9c54e0286554def5a3a6e3acd8428eabb33" gracePeriod=45
Sep 09 04:24:32.044012 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.043952    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:24:32.172473 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.172427    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj5hk\" (UniqueName: \"kubernetes.io/projected/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-kube-api-access-jj5hk\") pod \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") "
Sep 09 04:24:32.172595 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.172498    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-dshm\") pod \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") "
Sep 09 04:24:32.172595 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.172551    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-model-cache\") pod \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") "
Sep 09 04:24:32.172595 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.172574    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-tls-certs\") pod \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") "
Sep 09 04:24:32.172756 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.172607    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-home\") pod \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") "
Sep 09 04:24:32.172756 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.172635    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-tmp-dir\") pod \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\" (UID: \"fbaeba7e-5d6f-452b-966a-2e7c1054a17a\") "
Sep 09 04:24:32.172878 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.172849    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-model-cache" (OuterVolumeSpecName: "model-cache") pod "fbaeba7e-5d6f-452b-966a-2e7c1054a17a" (UID: "fbaeba7e-5d6f-452b-966a-2e7c1054a17a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:32.172878 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.172859    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-home" (OuterVolumeSpecName: "home") pod "fbaeba7e-5d6f-452b-966a-2e7c1054a17a" (UID: "fbaeba7e-5d6f-452b-966a-2e7c1054a17a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:32.173005 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.172980    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fbaeba7e-5d6f-452b-966a-2e7c1054a17a" (UID: "fbaeba7e-5d6f-452b-966a-2e7c1054a17a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:32.174868 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.174848    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fbaeba7e-5d6f-452b-966a-2e7c1054a17a" (UID: "fbaeba7e-5d6f-452b-966a-2e7c1054a17a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:24:32.175076 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.175059    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-dshm" (OuterVolumeSpecName: "dshm") pod "fbaeba7e-5d6f-452b-966a-2e7c1054a17a" (UID: "fbaeba7e-5d6f-452b-966a-2e7c1054a17a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:32.175135 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.175061    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-kube-api-access-jj5hk" (OuterVolumeSpecName: "kube-api-access-jj5hk") pod "fbaeba7e-5d6f-452b-966a-2e7c1054a17a" (UID: "fbaeba7e-5d6f-452b-966a-2e7c1054a17a"). InnerVolumeSpecName "kube-api-access-jj5hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:24:32.273588 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.273565    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:32.273588 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.273587    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:32.273717 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.273597    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj5hk\" (UniqueName: \"kubernetes.io/projected/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-kube-api-access-jj5hk\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:32.273717 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.273605    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:32.273717 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.273613    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:32.273717 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.273621    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbaeba7e-5d6f-452b-966a-2e7c1054a17a-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:32.471741 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.471711    2553 generic.go:356] "Generic (PLEG): container finished" podID="fbaeba7e-5d6f-452b-966a-2e7c1054a17a" containerID="3456d3a50f8b32b6ad3a2a7c286fd9c54e0286554def5a3a6e3acd8428eabb33" exitCode=0
Sep 09 04:24:32.471880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.471758    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"
Sep 09 04:24:32.471880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.471766    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz" event={"ID":"fbaeba7e-5d6f-452b-966a-2e7c1054a17a","Type":"ContainerDied","Data":"3456d3a50f8b32b6ad3a2a7c286fd9c54e0286554def5a3a6e3acd8428eabb33"}
Sep 09 04:24:32.471880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.471802    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz" event={"ID":"fbaeba7e-5d6f-452b-966a-2e7c1054a17a","Type":"ContainerDied","Data":"b4def1f7e7b620aa5d80b872ce9460bb7f54cd5654f47e8384a5ea28a6232707"}
Sep 09 04:24:32.471880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.471826    2553 scope.go:117] "RemoveContainer" containerID="3456d3a50f8b32b6ad3a2a7c286fd9c54e0286554def5a3a6e3acd8428eabb33"
Sep 09 04:24:32.489480 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.489432    2553 scope.go:117] "RemoveContainer" containerID="3456d3a50f8b32b6ad3a2a7c286fd9c54e0286554def5a3a6e3acd8428eabb33"
Sep 09 04:24:32.489736 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:24:32.489714    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3456d3a50f8b32b6ad3a2a7c286fd9c54e0286554def5a3a6e3acd8428eabb33\": container with ID starting with 3456d3a50f8b32b6ad3a2a7c286fd9c54e0286554def5a3a6e3acd8428eabb33 not found: ID does not exist" containerID="3456d3a50f8b32b6ad3a2a7c286fd9c54e0286554def5a3a6e3acd8428eabb33"
Sep 09 04:24:32.489784 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.489745    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3456d3a50f8b32b6ad3a2a7c286fd9c54e0286554def5a3a6e3acd8428eabb33"} err="failed to get container status \"3456d3a50f8b32b6ad3a2a7c286fd9c54e0286554def5a3a6e3acd8428eabb33\": rpc error: code = NotFound desc = could not find container \"3456d3a50f8b32b6ad3a2a7c286fd9c54e0286554def5a3a6e3acd8428eabb33\": container with ID starting with 3456d3a50f8b32b6ad3a2a7c286fd9c54e0286554def5a3a6e3acd8428eabb33 not found: ID does not exist"
Sep 09 04:24:32.499240 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.499218    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"]
Sep 09 04:24:32.502791 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.502772    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-426183/auth-test-model-426183-kserve-cb799f6cf-vjnmz"]
Sep 09 04:24:32.989471 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:32.989425    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbaeba7e-5d6f-452b-966a-2e7c1054a17a" path="/var/lib/kubelet/pods/fbaeba7e-5d6f-452b-966a-2e7c1054a17a/volumes"
Sep 09 04:24:33.372577 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.372489    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx" podUID="704f7ad4-59be-40a8-90de-c869d9260f34" containerName="main" containerID="cri-o://b8d47a98b905b6387eaea25078fc7e1498c1abb8edc6e4218098267d329e8f6b" gracePeriod=45
Sep 09 04:24:33.609505 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.609480    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:24:33.787767 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.787737    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-model-cache\") pod \"704f7ad4-59be-40a8-90de-c869d9260f34\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") "
Sep 09 04:24:33.787916 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.787798    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704f7ad4-59be-40a8-90de-c869d9260f34-tls-certs\") pod \"704f7ad4-59be-40a8-90de-c869d9260f34\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") "
Sep 09 04:24:33.787916 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.787829    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-dshm\") pod \"704f7ad4-59be-40a8-90de-c869d9260f34\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") "
Sep 09 04:24:33.787916 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.787860    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-tmp-dir\") pod \"704f7ad4-59be-40a8-90de-c869d9260f34\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") "
Sep 09 04:24:33.788069 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.787945    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-home\") pod \"704f7ad4-59be-40a8-90de-c869d9260f34\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") "
Sep 09 04:24:33.788069 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.787972    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8929q\" (UniqueName: \"kubernetes.io/projected/704f7ad4-59be-40a8-90de-c869d9260f34-kube-api-access-8929q\") pod \"704f7ad4-59be-40a8-90de-c869d9260f34\" (UID: \"704f7ad4-59be-40a8-90de-c869d9260f34\") "
Sep 09 04:24:33.788069 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.788027    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-model-cache" (OuterVolumeSpecName: "model-cache") pod "704f7ad4-59be-40a8-90de-c869d9260f34" (UID: "704f7ad4-59be-40a8-90de-c869d9260f34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:33.788224 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.788199    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "704f7ad4-59be-40a8-90de-c869d9260f34" (UID: "704f7ad4-59be-40a8-90de-c869d9260f34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:33.788270 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.788212    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-home" (OuterVolumeSpecName: "home") pod "704f7ad4-59be-40a8-90de-c869d9260f34" (UID: "704f7ad4-59be-40a8-90de-c869d9260f34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:33.788376 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.788356    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:33.788376 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.788372    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:33.788376 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.788383    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:33.790347 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.790322    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704f7ad4-59be-40a8-90de-c869d9260f34-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "704f7ad4-59be-40a8-90de-c869d9260f34" (UID: "704f7ad4-59be-40a8-90de-c869d9260f34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:24:33.790496 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.790353    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-dshm" (OuterVolumeSpecName: "dshm") pod "704f7ad4-59be-40a8-90de-c869d9260f34" (UID: "704f7ad4-59be-40a8-90de-c869d9260f34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:33.790496 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.790465    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704f7ad4-59be-40a8-90de-c869d9260f34-kube-api-access-8929q" (OuterVolumeSpecName: "kube-api-access-8929q") pod "704f7ad4-59be-40a8-90de-c869d9260f34" (UID: "704f7ad4-59be-40a8-90de-c869d9260f34"). InnerVolumeSpecName "kube-api-access-8929q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:24:33.889021 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.888998    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704f7ad4-59be-40a8-90de-c869d9260f34-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:33.889021 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.889020    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704f7ad4-59be-40a8-90de-c869d9260f34-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:33.889150 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.889029    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8929q\" (UniqueName: \"kubernetes.io/projected/704f7ad4-59be-40a8-90de-c869d9260f34-kube-api-access-8929q\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:24:33.966777 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:33.966750    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:24:34.485545 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:34.485510    2553 generic.go:356] "Generic (PLEG): container finished" podID="704f7ad4-59be-40a8-90de-c869d9260f34" containerID="b8d47a98b905b6387eaea25078fc7e1498c1abb8edc6e4218098267d329e8f6b" exitCode=0
Sep 09 04:24:34.485925 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:34.485567    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"
Sep 09 04:24:34.485925 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:34.485589    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx" event={"ID":"704f7ad4-59be-40a8-90de-c869d9260f34","Type":"ContainerDied","Data":"b8d47a98b905b6387eaea25078fc7e1498c1abb8edc6e4218098267d329e8f6b"}
Sep 09 04:24:34.485925 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:34.485633    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx" event={"ID":"704f7ad4-59be-40a8-90de-c869d9260f34","Type":"ContainerDied","Data":"5ddbd0337a9afdd1e0ac07f4e9ae4f09698d384904d853e719190ad985b47e1b"}
Sep 09 04:24:34.485925 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:34.485653    2553 scope.go:117] "RemoveContainer" containerID="b8d47a98b905b6387eaea25078fc7e1498c1abb8edc6e4218098267d329e8f6b"
Sep 09 04:24:34.494849 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:34.494831    2553 scope.go:117] "RemoveContainer" containerID="b8d47a98b905b6387eaea25078fc7e1498c1abb8edc6e4218098267d329e8f6b"
Sep 09 04:24:34.495114 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:24:34.495096    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d47a98b905b6387eaea25078fc7e1498c1abb8edc6e4218098267d329e8f6b\": container with ID starting with b8d47a98b905b6387eaea25078fc7e1498c1abb8edc6e4218098267d329e8f6b not found: ID does not exist" containerID="b8d47a98b905b6387eaea25078fc7e1498c1abb8edc6e4218098267d329e8f6b"
Sep 09 04:24:34.495167 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:34.495122    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d47a98b905b6387eaea25078fc7e1498c1abb8edc6e4218098267d329e8f6b"} err="failed to get container status \"b8d47a98b905b6387eaea25078fc7e1498c1abb8edc6e4218098267d329e8f6b\": rpc error: code = NotFound desc = could not find container \"b8d47a98b905b6387eaea25078fc7e1498c1abb8edc6e4218098267d329e8f6b\": container with ID starting with b8d47a98b905b6387eaea25078fc7e1498c1abb8edc6e4218098267d329e8f6b not found: ID does not exist"
Sep 09 04:24:34.506752 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:34.506733    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"]
Sep 09 04:24:34.510452 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:34.510419    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/auth-test-model-9c44b5-kserve-79cc44dc5f-252tx"]
Sep 09 04:24:34.989354 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:34.989306    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="704f7ad4-59be-40a8-90de-c869d9260f34" path="/var/lib/kubelet/pods/704f7ad4-59be-40a8-90de-c869d9260f34/volumes"
Sep 09 04:24:36.964829 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.964800    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"]
Sep 09 04:24:36.965234 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965221    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55596728-74e8-4c54-ba37-f930bf20e6d0" containerName="main"
Sep 09 04:24:36.965234 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965235    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="55596728-74e8-4c54-ba37-f930bf20e6d0" containerName="main"
Sep 09 04:24:36.965314 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965242    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="162a095d-833a-45fb-adc1-a6b77a44a1ca" containerName="revoke-keys"
Sep 09 04:24:36.965314 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965249    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="162a095d-833a-45fb-adc1-a6b77a44a1ca" containerName="revoke-keys"
Sep 09 04:24:36.965314 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965255    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393d7150-22f6-48a0-a718-c590930bea06" containerName="payload-processing"
Sep 09 04:24:36.965314 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965260    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="393d7150-22f6-48a0-a718-c590930bea06" containerName="payload-processing"
Sep 09 04:24:36.965314 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965277    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="197d60c8-9bf1-4af7-9739-857ae3d05438" containerName="main"
Sep 09 04:24:36.965314 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965282    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="197d60c8-9bf1-4af7-9739-857ae3d05438" containerName="main"
Sep 09 04:24:36.965314 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965300    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704f7ad4-59be-40a8-90de-c869d9260f34" containerName="main"
Sep 09 04:24:36.965314 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965306    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="704f7ad4-59be-40a8-90de-c869d9260f34" containerName="main"
Sep 09 04:24:36.965314 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965316    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbaeba7e-5d6f-452b-966a-2e7c1054a17a" containerName="main"
Sep 09 04:24:36.965630 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965320    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbaeba7e-5d6f-452b-966a-2e7c1054a17a" containerName="main"
Sep 09 04:24:36.965630 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965354    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b02176de-51f1-4f4a-8b93-e4008228f719" containerName="main"
Sep 09 04:24:36.965630 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965359    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b02176de-51f1-4f4a-8b93-e4008228f719" containerName="main"
Sep 09 04:24:36.965630 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965412    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="197d60c8-9bf1-4af7-9739-857ae3d05438" containerName="main"
Sep 09 04:24:36.965630 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965420    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="55596728-74e8-4c54-ba37-f930bf20e6d0" containerName="main"
Sep 09 04:24:36.965630 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965426    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b02176de-51f1-4f4a-8b93-e4008228f719" containerName="main"
Sep 09 04:24:36.965630 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965432    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="704f7ad4-59be-40a8-90de-c869d9260f34" containerName="main"
Sep 09 04:24:36.965630 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965461    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="162a095d-833a-45fb-adc1-a6b77a44a1ca" containerName="revoke-keys"
Sep 09 04:24:36.965630 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965469    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="393d7150-22f6-48a0-a718-c590930bea06" containerName="payload-processing"
Sep 09 04:24:36.965630 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.965477    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbaeba7e-5d6f-452b-966a-2e7c1054a17a" containerName="main"
Sep 09 04:24:36.969776 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.969755    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:36.973268 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.973246    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-426183\"/\"default-dockercfg-q6z8l\""
Sep 09 04:24:36.973268 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.973258    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-426183\"/\"sub-test-model-426183-kserve-self-signed-certs\""
Sep 09 04:24:36.973522 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.973246    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-426183\"/\"kube-root-ca.crt\""
Sep 09 04:24:36.973522 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.973349    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-426183\"/\"openshift-service-ca.crt\""
Sep 09 04:24:36.978616 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:36.978592    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"]
Sep 09 04:24:37.086190 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.086158    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:37.117964 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.117939    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-home\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.118084 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.118006    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-dshm\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.118084 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.118030    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-model-cache\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.118084 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.118060    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604f5521-ae05-4f4e-941b-81ce3d98ed55-tls-certs\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.118191 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.118084    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqcvt\" (UniqueName: \"kubernetes.io/projected/604f5521-ae05-4f4e-941b-81ce3d98ed55-kube-api-access-dqcvt\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.118191 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.118108    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-tmp-dir\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.218569 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.218506    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-home\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.218569 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.218542    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-dshm\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.218569 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.218562    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-model-cache\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.218747 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.218584    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604f5521-ae05-4f4e-941b-81ce3d98ed55-tls-certs\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.218747 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.218614    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqcvt\" (UniqueName: \"kubernetes.io/projected/604f5521-ae05-4f4e-941b-81ce3d98ed55-kube-api-access-dqcvt\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.218747 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.218641    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-tmp-dir\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.218939 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.218918    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-home\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.219030 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.218935    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-model-cache\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.219119 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.219101    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-tmp-dir\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.220998 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.220974    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-dshm\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.221375 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.221354    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604f5521-ae05-4f4e-941b-81ce3d98ed55-tls-certs\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.229873 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.229853    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqcvt\" (UniqueName: \"kubernetes.io/projected/604f5521-ae05-4f4e-941b-81ce3d98ed55-kube-api-access-dqcvt\") pod \"sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") " pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.280563 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.280545    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:37.554499 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:37.554468    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"]
Sep 09 04:24:37.555808 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:24:37.555780    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod604f5521_ae05_4f4e_941b_81ce3d98ed55.slice/crio-4bdb90fc7a913a23fa713e3ca524d340c3b5c68b2df7a87b09ef043825d8b46e WatchSource:0}: Error finding container 4bdb90fc7a913a23fa713e3ca524d340c3b5c68b2df7a87b09ef043825d8b46e: Status 404 returned error can't find the container with id 4bdb90fc7a913a23fa713e3ca524d340c3b5c68b2df7a87b09ef043825d8b46e
Sep 09 04:24:38.509977 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:38.509944    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v" event={"ID":"604f5521-ae05-4f4e-941b-81ce3d98ed55","Type":"ContainerStarted","Data":"b492dc67928ca6ef332fa3fd1b2d0bcd7b8bc45d21de4426d34ccd2451033ac9"}
Sep 09 04:24:38.510428 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:38.509988    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v" event={"ID":"604f5521-ae05-4f4e-941b-81ce3d98ed55","Type":"ContainerStarted","Data":"4bdb90fc7a913a23fa713e3ca524d340c3b5c68b2df7a87b09ef043825d8b46e"}
Sep 09 04:24:38.528743 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:38.528702    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v" podStartSLOduration=2.528688386 podStartE2EDuration="2.528688386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:24:38.526482381 +0000 UTC m=+1578.052853092" watchObservedRunningTime="2026-09-09 04:24:38.528688386 +0000 UTC m=+1578.055059087"
Sep 09 04:24:43.915319 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:43.915288    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:44.010627 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:44.010589    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:44.798773 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:44.798741    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:47.281275 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:47.281243    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:47.281705 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:47.281300    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:47.294264 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:47.294242    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:47.568345 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:47.568270    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:24:50.793680 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:50.793645    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:24:54.159682 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:24:54.159652    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:25:05.531901 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:05.531867    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:25:05.631477 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:05.631393    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:25:08.909828 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:08.909798    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:25:28.863890 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:28.863854    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"]
Sep 09 04:25:28.868913 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:28.868896    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:28.872405 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:28.872382    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9c44b5\"/\"kube-root-ca.crt\""
Sep 09 04:25:28.872405 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:28.872396    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9c44b5\"/\"openshift-service-ca.crt\""
Sep 09 04:25:28.872627 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:28.872382    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9c44b5\"/\"default-dockercfg-jdl5d\""
Sep 09 04:25:28.872627 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:28.872381    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9c44b5\"/\"sub-test-model-9c44b5-kserve-self-signed-certs\""
Sep 09 04:25:28.877423 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:28.877403    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"]
Sep 09 04:25:28.892797 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:28.892759    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:25:28.966868 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:28.966848    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-tmp-dir\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:28.966979 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:28.966875    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs7lh\" (UniqueName: \"kubernetes.io/projected/119ad916-3c23-40da-bd04-6cd707ce1f5d-kube-api-access-zs7lh\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:28.966979 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:28.966906    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-home\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:28.966979 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:28.966920    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-model-cache\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:28.967081 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:28.966980    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119ad916-3c23-40da-bd04-6cd707ce1f5d-tls-certs\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:28.967081 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:28.967064    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-dshm\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:29.067929 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.067899    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-home\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:29.067929 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.067934    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-model-cache\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:29.068111 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.067959    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119ad916-3c23-40da-bd04-6cd707ce1f5d-tls-certs\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:29.068111 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.068012    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-dshm\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:29.068111 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.068082    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-tmp-dir\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:29.068111 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.068106    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs7lh\" (UniqueName: \"kubernetes.io/projected/119ad916-3c23-40da-bd04-6cd707ce1f5d-kube-api-access-zs7lh\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:29.068320 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.068279    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-home\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:29.068376 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.068331    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-model-cache\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:29.068460 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.068424    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-tmp-dir\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:29.070557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.070535    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-dshm\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:29.070755 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.070741    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119ad916-3c23-40da-bd04-6cd707ce1f5d-tls-certs\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:29.076651 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.076634    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs7lh\" (UniqueName: \"kubernetes.io/projected/119ad916-3c23-40da-bd04-6cd707ce1f5d-kube-api-access-zs7lh\") pod \"sub-test-model-9c44b5-kserve-659d5b8585-bdlqx\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:29.179351 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.179294    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:29.450460 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.450405    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"]
Sep 09 04:25:29.453987 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:25:29.453952    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod119ad916_3c23_40da_bd04_6cd707ce1f5d.slice/crio-5173e6d924db8e3231512078476a1d3489130dd698b5e9d10ce800640541e364 WatchSource:0}: Error finding container 5173e6d924db8e3231512078476a1d3489130dd698b5e9d10ce800640541e364: Status 404 returned error can't find the container with id 5173e6d924db8e3231512078476a1d3489130dd698b5e9d10ce800640541e364
Sep 09 04:25:29.797050 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.797018    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx" event={"ID":"119ad916-3c23-40da-bd04-6cd707ce1f5d","Type":"ContainerStarted","Data":"c67686fa9b2501e20f6991702e8b9c63202b5b47c2f1fd61e0b4977fee97a086"}
Sep 09 04:25:29.797050 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.797056    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx" event={"ID":"119ad916-3c23-40da-bd04-6cd707ce1f5d","Type":"ContainerStarted","Data":"5173e6d924db8e3231512078476a1d3489130dd698b5e9d10ce800640541e364"}
Sep 09 04:25:29.820969 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:29.820924    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx" podStartSLOduration=1.82091038 podStartE2EDuration="1.82091038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:25:29.819112619 +0000 UTC m=+1629.345483340" watchObservedRunningTime="2026-09-09 04:25:29.82091038 +0000 UTC m=+1629.347281089"
Sep 09 04:25:33.369876 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:33.369841    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:25:39.180069 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:39.180039    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:39.180490 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:39.180280    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:39.192557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:39.192538    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:39.867252 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:39.867223    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:25:48.959284 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:25:48.959244    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:26:03.765509 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:03.765472    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:26:06.218967 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:06.218934    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:26:15.169042 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:15.169005    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"]
Sep 09 04:26:15.198642 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:15.198611    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:26:16.444362 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:16.444330    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"]
Sep 09 04:26:16.457929 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:16.457904    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:26:18.267941 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:18.267904    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:26:21.324330 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:21.324293    2553 scope.go:117] "RemoveContainer" containerID="264b108a3e0e62d92209377c80dd672f972558120148d364cf44f5b43f52afb8"
Sep 09 04:26:26.198680 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:26.198651    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:26:29.777371 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:29.777337    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:26:30.188583 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.188494    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v" podUID="604f5521-ae05-4f4e-941b-81ce3d98ed55" containerName="main" containerID="cri-o://b492dc67928ca6ef332fa3fd1b2d0bcd7b8bc45d21de4426d34ccd2451033ac9" gracePeriod=45
Sep 09 04:26:30.441508 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.441430    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:26:30.511864 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.511841    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-tmp-dir\") pod \"604f5521-ae05-4f4e-941b-81ce3d98ed55\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") "
Sep 09 04:26:30.511991 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.511875    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqcvt\" (UniqueName: \"kubernetes.io/projected/604f5521-ae05-4f4e-941b-81ce3d98ed55-kube-api-access-dqcvt\") pod \"604f5521-ae05-4f4e-941b-81ce3d98ed55\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") "
Sep 09 04:26:30.511991 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.511892    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604f5521-ae05-4f4e-941b-81ce3d98ed55-tls-certs\") pod \"604f5521-ae05-4f4e-941b-81ce3d98ed55\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") "
Sep 09 04:26:30.511991 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.511938    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-dshm\") pod \"604f5521-ae05-4f4e-941b-81ce3d98ed55\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") "
Sep 09 04:26:30.512136 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.512017    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-home\") pod \"604f5521-ae05-4f4e-941b-81ce3d98ed55\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") "
Sep 09 04:26:30.512136 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.512036    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-model-cache\") pod \"604f5521-ae05-4f4e-941b-81ce3d98ed55\" (UID: \"604f5521-ae05-4f4e-941b-81ce3d98ed55\") "
Sep 09 04:26:30.512136 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.512083    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "604f5521-ae05-4f4e-941b-81ce3d98ed55" (UID: "604f5521-ae05-4f4e-941b-81ce3d98ed55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:26:30.512342 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.512316    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-home" (OuterVolumeSpecName: "home") pod "604f5521-ae05-4f4e-941b-81ce3d98ed55" (UID: "604f5521-ae05-4f4e-941b-81ce3d98ed55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:26:30.512421 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.512329    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:26:30.512421 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.512347    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-model-cache" (OuterVolumeSpecName: "model-cache") pod "604f5521-ae05-4f4e-941b-81ce3d98ed55" (UID: "604f5521-ae05-4f4e-941b-81ce3d98ed55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:26:30.514145 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.514122    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604f5521-ae05-4f4e-941b-81ce3d98ed55-kube-api-access-dqcvt" (OuterVolumeSpecName: "kube-api-access-dqcvt") pod "604f5521-ae05-4f4e-941b-81ce3d98ed55" (UID: "604f5521-ae05-4f4e-941b-81ce3d98ed55"). InnerVolumeSpecName "kube-api-access-dqcvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:26:30.514245 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.514139    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-dshm" (OuterVolumeSpecName: "dshm") pod "604f5521-ae05-4f4e-941b-81ce3d98ed55" (UID: "604f5521-ae05-4f4e-941b-81ce3d98ed55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:26:30.514245 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.514229    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604f5521-ae05-4f4e-941b-81ce3d98ed55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "604f5521-ae05-4f4e-941b-81ce3d98ed55" (UID: "604f5521-ae05-4f4e-941b-81ce3d98ed55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:26:30.613626 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.613599    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:26:30.613626 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.613621    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:26:30.613626 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.613631    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqcvt\" (UniqueName: \"kubernetes.io/projected/604f5521-ae05-4f4e-941b-81ce3d98ed55-kube-api-access-dqcvt\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:26:30.613842 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.613639    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604f5521-ae05-4f4e-941b-81ce3d98ed55-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:26:30.613842 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:30.613647    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604f5521-ae05-4f4e-941b-81ce3d98ed55-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:26:31.118736 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.118705    2553 generic.go:356] "Generic (PLEG): container finished" podID="604f5521-ae05-4f4e-941b-81ce3d98ed55" containerID="b492dc67928ca6ef332fa3fd1b2d0bcd7b8bc45d21de4426d34ccd2451033ac9" exitCode=0
Sep 09 04:26:31.119099 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.118741    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"
Sep 09 04:26:31.119099 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.118783    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v" event={"ID":"604f5521-ae05-4f4e-941b-81ce3d98ed55","Type":"ContainerDied","Data":"b492dc67928ca6ef332fa3fd1b2d0bcd7b8bc45d21de4426d34ccd2451033ac9"}
Sep 09 04:26:31.119099 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.118817    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v" event={"ID":"604f5521-ae05-4f4e-941b-81ce3d98ed55","Type":"ContainerDied","Data":"4bdb90fc7a913a23fa713e3ca524d340c3b5c68b2df7a87b09ef043825d8b46e"}
Sep 09 04:26:31.119099 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.118835    2553 scope.go:117] "RemoveContainer" containerID="b492dc67928ca6ef332fa3fd1b2d0bcd7b8bc45d21de4426d34ccd2451033ac9"
Sep 09 04:26:31.127601 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.127576    2553 scope.go:117] "RemoveContainer" containerID="b492dc67928ca6ef332fa3fd1b2d0bcd7b8bc45d21de4426d34ccd2451033ac9"
Sep 09 04:26:31.127891 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:26:31.127872    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b492dc67928ca6ef332fa3fd1b2d0bcd7b8bc45d21de4426d34ccd2451033ac9\": container with ID starting with b492dc67928ca6ef332fa3fd1b2d0bcd7b8bc45d21de4426d34ccd2451033ac9 not found: ID does not exist" containerID="b492dc67928ca6ef332fa3fd1b2d0bcd7b8bc45d21de4426d34ccd2451033ac9"
Sep 09 04:26:31.127944 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.127899    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b492dc67928ca6ef332fa3fd1b2d0bcd7b8bc45d21de4426d34ccd2451033ac9"} err="failed to get container status \"b492dc67928ca6ef332fa3fd1b2d0bcd7b8bc45d21de4426d34ccd2451033ac9\": rpc error: code = NotFound desc = could not find container \"b492dc67928ca6ef332fa3fd1b2d0bcd7b8bc45d21de4426d34ccd2451033ac9\": container with ID starting with b492dc67928ca6ef332fa3fd1b2d0bcd7b8bc45d21de4426d34ccd2451033ac9 not found: ID does not exist"
Sep 09 04:26:31.137877 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.137859    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"]
Sep 09 04:26:31.143813 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.143793    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-426183/sub-test-model-426183-kserve-6f9bdcbfd7-8pd9v"]
Sep 09 04:26:31.464176 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.464107    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx" podUID="119ad916-3c23-40da-bd04-6cd707ce1f5d" containerName="main" containerID="cri-o://c67686fa9b2501e20f6991702e8b9c63202b5b47c2f1fd61e0b4977fee97a086" gracePeriod=45
Sep 09 04:26:31.705767 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.705744    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:26:31.822541 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.822517    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119ad916-3c23-40da-bd04-6cd707ce1f5d-tls-certs\") pod \"119ad916-3c23-40da-bd04-6cd707ce1f5d\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") "
Sep 09 04:26:31.822676 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.822585    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-home\") pod \"119ad916-3c23-40da-bd04-6cd707ce1f5d\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") "
Sep 09 04:26:31.822676 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.822599    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-dshm\") pod \"119ad916-3c23-40da-bd04-6cd707ce1f5d\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") "
Sep 09 04:26:31.822792 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.822708    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-model-cache\") pod \"119ad916-3c23-40da-bd04-6cd707ce1f5d\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") "
Sep 09 04:26:31.822792 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.822782    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-tmp-dir\") pod \"119ad916-3c23-40da-bd04-6cd707ce1f5d\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") "
Sep 09 04:26:31.822898 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.822824    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs7lh\" (UniqueName: \"kubernetes.io/projected/119ad916-3c23-40da-bd04-6cd707ce1f5d-kube-api-access-zs7lh\") pod \"119ad916-3c23-40da-bd04-6cd707ce1f5d\" (UID: \"119ad916-3c23-40da-bd04-6cd707ce1f5d\") "
Sep 09 04:26:31.822898 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.822854    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-home" (OuterVolumeSpecName: "home") pod "119ad916-3c23-40da-bd04-6cd707ce1f5d" (UID: "119ad916-3c23-40da-bd04-6cd707ce1f5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:26:31.823015 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.822937    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "119ad916-3c23-40da-bd04-6cd707ce1f5d" (UID: "119ad916-3c23-40da-bd04-6cd707ce1f5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:26:31.823059 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.823011    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "119ad916-3c23-40da-bd04-6cd707ce1f5d" (UID: "119ad916-3c23-40da-bd04-6cd707ce1f5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:26:31.823265 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.823249    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:26:31.823305 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.823272    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:26:31.823305 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.823286    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:26:31.824783 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.824762    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/119ad916-3c23-40da-bd04-6cd707ce1f5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "119ad916-3c23-40da-bd04-6cd707ce1f5d" (UID: "119ad916-3c23-40da-bd04-6cd707ce1f5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:26:31.824783 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.824768    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-dshm" (OuterVolumeSpecName: "dshm") pod "119ad916-3c23-40da-bd04-6cd707ce1f5d" (UID: "119ad916-3c23-40da-bd04-6cd707ce1f5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:26:31.824940 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.824881    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119ad916-3c23-40da-bd04-6cd707ce1f5d-kube-api-access-zs7lh" (OuterVolumeSpecName: "kube-api-access-zs7lh") pod "119ad916-3c23-40da-bd04-6cd707ce1f5d" (UID: "119ad916-3c23-40da-bd04-6cd707ce1f5d"). InnerVolumeSpecName "kube-api-access-zs7lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:26:31.924607 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.924588    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119ad916-3c23-40da-bd04-6cd707ce1f5d-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:26:31.924607 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.924605    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119ad916-3c23-40da-bd04-6cd707ce1f5d-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:26:31.924739 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:31.924614    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs7lh\" (UniqueName: \"kubernetes.io/projected/119ad916-3c23-40da-bd04-6cd707ce1f5d-kube-api-access-zs7lh\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:26:32.125110 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:32.125042    2553 generic.go:356] "Generic (PLEG): container finished" podID="119ad916-3c23-40da-bd04-6cd707ce1f5d" containerID="c67686fa9b2501e20f6991702e8b9c63202b5b47c2f1fd61e0b4977fee97a086" exitCode=0
Sep 09 04:26:32.125110 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:32.125080    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx" event={"ID":"119ad916-3c23-40da-bd04-6cd707ce1f5d","Type":"ContainerDied","Data":"c67686fa9b2501e20f6991702e8b9c63202b5b47c2f1fd61e0b4977fee97a086"}
Sep 09 04:26:32.125570 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:32.125115    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"
Sep 09 04:26:32.125570 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:32.125129    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx" event={"ID":"119ad916-3c23-40da-bd04-6cd707ce1f5d","Type":"ContainerDied","Data":"5173e6d924db8e3231512078476a1d3489130dd698b5e9d10ce800640541e364"}
Sep 09 04:26:32.125570 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:32.125150    2553 scope.go:117] "RemoveContainer" containerID="c67686fa9b2501e20f6991702e8b9c63202b5b47c2f1fd61e0b4977fee97a086"
Sep 09 04:26:32.134520 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:32.134504    2553 scope.go:117] "RemoveContainer" containerID="c67686fa9b2501e20f6991702e8b9c63202b5b47c2f1fd61e0b4977fee97a086"
Sep 09 04:26:32.134773 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:26:32.134755    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67686fa9b2501e20f6991702e8b9c63202b5b47c2f1fd61e0b4977fee97a086\": container with ID starting with c67686fa9b2501e20f6991702e8b9c63202b5b47c2f1fd61e0b4977fee97a086 not found: ID does not exist" containerID="c67686fa9b2501e20f6991702e8b9c63202b5b47c2f1fd61e0b4977fee97a086"
Sep 09 04:26:32.134826 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:32.134781    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67686fa9b2501e20f6991702e8b9c63202b5b47c2f1fd61e0b4977fee97a086"} err="failed to get container status \"c67686fa9b2501e20f6991702e8b9c63202b5b47c2f1fd61e0b4977fee97a086\": rpc error: code = NotFound desc = could not find container \"c67686fa9b2501e20f6991702e8b9c63202b5b47c2f1fd61e0b4977fee97a086\": container with ID starting with c67686fa9b2501e20f6991702e8b9c63202b5b47c2f1fd61e0b4977fee97a086 not found: ID does not exist"
Sep 09 04:26:32.146769 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:32.146745    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"]
Sep 09 04:26:32.153524 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:32.153503    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/sub-test-model-9c44b5-kserve-659d5b8585-bdlqx"]
Sep 09 04:26:32.989990 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:32.989962    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="119ad916-3c23-40da-bd04-6cd707ce1f5d" path="/var/lib/kubelet/pods/119ad916-3c23-40da-bd04-6cd707ce1f5d/volumes"
Sep 09 04:26:32.990339 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:26:32.990325    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="604f5521-ae05-4f4e-941b-81ce3d98ed55" path="/var/lib/kubelet/pods/604f5521-ae05-4f4e-941b-81ce3d98ed55/volumes"
Sep 09 04:27:03.168667 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.168629    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"]
Sep 09 04:27:03.169212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.169052    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="119ad916-3c23-40da-bd04-6cd707ce1f5d" containerName="main"
Sep 09 04:27:03.169212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.169063    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="119ad916-3c23-40da-bd04-6cd707ce1f5d" containerName="main"
Sep 09 04:27:03.169212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.169077    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="604f5521-ae05-4f4e-941b-81ce3d98ed55" containerName="main"
Sep 09 04:27:03.169212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.169081    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="604f5521-ae05-4f4e-941b-81ce3d98ed55" containerName="main"
Sep 09 04:27:03.169212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.169151    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="119ad916-3c23-40da-bd04-6cd707ce1f5d" containerName="main"
Sep 09 04:27:03.169212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.169160    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="604f5521-ae05-4f4e-941b-81ce3d98ed55" containerName="main"
Sep 09 04:27:03.172385 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.172362    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.175709 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.175688    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9c44b5\"/\"default-dockercfg-jdl5d\""
Sep 09 04:27:03.175936 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.175708    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9c44b5\"/\"kube-root-ca.crt\""
Sep 09 04:27:03.176107 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.175737    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9c44b5\"/\"rate-test-model-9c44b5-kserve-self-signed-certs\""
Sep 09 04:27:03.176227 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.175791    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9c44b5\"/\"openshift-service-ca.crt\""
Sep 09 04:27:03.178820 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.178799    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"]
Sep 09 04:27:03.189192 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.189164    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:03.195321 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.195299    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-dshm\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.195426 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.195336    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn4vq\" (UniqueName: \"kubernetes.io/projected/247f310d-7fdf-47e5-a478-f8afeed670b7-kube-api-access-hn4vq\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.195426 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.195408    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-tmp-dir\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.195581 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.195437    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247f310d-7fdf-47e5-a478-f8afeed670b7-tls-certs\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.195581 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.195491    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-home\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.195581 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.195512    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-model-cache\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.296329 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.296305    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-dshm\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.296426 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.296340    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn4vq\" (UniqueName: \"kubernetes.io/projected/247f310d-7fdf-47e5-a478-f8afeed670b7-kube-api-access-hn4vq\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.296426 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.296388    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-tmp-dir\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.296426 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.296420    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247f310d-7fdf-47e5-a478-f8afeed670b7-tls-certs\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.296625 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.296477    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-home\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.296625 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.296504    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-model-cache\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.296756 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.296735    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-tmp-dir\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.296817 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.296778    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-home\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.296865 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.296848    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-model-cache\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.298662 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.298644    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-dshm\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.298931 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.298911    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247f310d-7fdf-47e5-a478-f8afeed670b7-tls-certs\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.306165 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.306143    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn4vq\" (UniqueName: \"kubernetes.io/projected/247f310d-7fdf-47e5-a478-f8afeed670b7-kube-api-access-hn4vq\") pod \"rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") " pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.483870 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.483843    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:03.754809 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:03.754783    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"]
Sep 09 04:27:03.756324 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:27:03.756299    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod247f310d_7fdf_47e5_a478_f8afeed670b7.slice/crio-5de156357f87747a8fc01f3b7977b7b2ccacdf2b79fb73a26ecd2559d00cd14c WatchSource:0}: Error finding container 5de156357f87747a8fc01f3b7977b7b2ccacdf2b79fb73a26ecd2559d00cd14c: Status 404 returned error can't find the container with id 5de156357f87747a8fc01f3b7977b7b2ccacdf2b79fb73a26ecd2559d00cd14c
Sep 09 04:27:04.290119 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:04.290086    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4" event={"ID":"247f310d-7fdf-47e5-a478-f8afeed670b7","Type":"ContainerStarted","Data":"1df4bd2c96f9e7c60c4a408477ebd5d320b91be110c7227b48dc175d3bc507e4"}
Sep 09 04:27:04.290520 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:04.290132    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4" event={"ID":"247f310d-7fdf-47e5-a478-f8afeed670b7","Type":"ContainerStarted","Data":"5de156357f87747a8fc01f3b7977b7b2ccacdf2b79fb73a26ecd2559d00cd14c"}
Sep 09 04:27:04.308158 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:04.308116    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4" podStartSLOduration=1.308102501 podStartE2EDuration="1.308102501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:27:04.306740342 +0000 UTC m=+1723.833111052" watchObservedRunningTime="2026-09-09 04:27:04.308102501 +0000 UTC m=+1723.834473209"
Sep 09 04:27:07.409221 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:07.409186    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:27:10.284351 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:10.284311    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:13.484132 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:13.484100    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:13.484599 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:13.484147    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:13.496355 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:13.496335    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:14.352152 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:14.352128    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:27:22.476468 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:22.474704    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:27:25.303018 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:25.302982    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:32.964202 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:32.964165    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:27:43.397206 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:43.397168    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:27:50.300636 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:50.300596    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:51.279774 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:51.279745    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"]
Sep 09 04:27:51.298802 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:51.298763    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:51.732596 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:51.732572    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:52.095997 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:52.095915    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:52.346898 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:52.346822    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:55.510845 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:55.510806    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:55.605475 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:55.605427    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:56.792651 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:56.792623    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:56.893519 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:56.893478    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:57.024465 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.024422    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:57.347529 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.347503    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:57.445650 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.445606    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:27:57.491353 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.491320    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"]
Sep 09 04:27:57.494934 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.494915    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"
Sep 09 04:27:57.497011 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.496986    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a3ce90-dockercfg-ggcz5\""
Sep 09 04:27:57.497124 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.497034    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a3ce90\""
Sep 09 04:27:57.506425 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.506406    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"]
Sep 09 04:27:57.561257 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.561229    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvgqq\" (UniqueName: \"kubernetes.io/projected/5b5d4fd5-d238-4dbd-8526-6a0c95762f39-kube-api-access-xvgqq\") pod \"payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz\" (UID: \"5b5d4fd5-d238-4dbd-8526-6a0c95762f39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"
Sep 09 04:27:57.561433 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.561395    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5d4fd5-d238-4dbd-8526-6a0c95762f39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz\" (UID: \"5b5d4fd5-d238-4dbd-8526-6a0c95762f39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"
Sep 09 04:27:57.662178 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.662107    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5d4fd5-d238-4dbd-8526-6a0c95762f39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz\" (UID: \"5b5d4fd5-d238-4dbd-8526-6a0c95762f39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"
Sep 09 04:27:57.662319 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.662193    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvgqq\" (UniqueName: \"kubernetes.io/projected/5b5d4fd5-d238-4dbd-8526-6a0c95762f39-kube-api-access-xvgqq\") pod \"payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz\" (UID: \"5b5d4fd5-d238-4dbd-8526-6a0c95762f39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"
Sep 09 04:27:57.662907 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.662887    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5d4fd5-d238-4dbd-8526-6a0c95762f39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz\" (UID: \"5b5d4fd5-d238-4dbd-8526-6a0c95762f39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"
Sep 09 04:27:57.674492 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.674467    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvgqq\" (UniqueName: \"kubernetes.io/projected/5b5d4fd5-d238-4dbd-8526-6a0c95762f39-kube-api-access-xvgqq\") pod \"payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz\" (UID: \"5b5d4fd5-d238-4dbd-8526-6a0c95762f39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"
Sep 09 04:27:57.806287 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:57.806263    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"
Sep 09 04:27:58.076936 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:58.076913    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"]
Sep 09 04:27:58.079139 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:27:58.079102    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5d4fd5_d238_4dbd_8526_6a0c95762f39.slice/crio-098d998fc90a2fdac804248515c5ed1e42f867783bc453a6aa78a80286003fb8 WatchSource:0}: Error finding container 098d998fc90a2fdac804248515c5ed1e42f867783bc453a6aa78a80286003fb8: Status 404 returned error can't find the container with id 098d998fc90a2fdac804248515c5ed1e42f867783bc453a6aa78a80286003fb8
Sep 09 04:27:58.572819 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:58.572782    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz" event={"ID":"5b5d4fd5-d238-4dbd-8526-6a0c95762f39","Type":"ContainerStarted","Data":"9e6c7174106c5c377cc28283c05ba278d3acd83cfe7d2d86f23de04135d136ca"}
Sep 09 04:27:58.573039 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:58.572857    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz" event={"ID":"5b5d4fd5-d238-4dbd-8526-6a0c95762f39","Type":"ContainerStarted","Data":"098d998fc90a2fdac804248515c5ed1e42f867783bc453a6aa78a80286003fb8"}
Sep 09 04:27:58.573039 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:58.572892    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"
Sep 09 04:27:58.590609 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:27:58.590572    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz" podStartSLOduration=1.59055738 podStartE2EDuration="1.59055738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:27:58.589338788 +0000 UTC m=+1778.115709498" watchObservedRunningTime="2026-09-09 04:27:58.59055738 +0000 UTC m=+1778.116928089"
Sep 09 04:28:01.870481 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:01.870436    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:28:06.299164 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.299101    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4" podUID="247f310d-7fdf-47e5-a478-f8afeed670b7" containerName="main" containerID="cri-o://1df4bd2c96f9e7c60c4a408477ebd5d320b91be110c7227b48dc175d3bc507e4" gracePeriod=45
Sep 09 04:28:06.543078 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.543053    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:28:06.614962 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.614890    2553 generic.go:356] "Generic (PLEG): container finished" podID="247f310d-7fdf-47e5-a478-f8afeed670b7" containerID="1df4bd2c96f9e7c60c4a408477ebd5d320b91be110c7227b48dc175d3bc507e4" exitCode=0
Sep 09 04:28:06.614962 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.614931    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"
Sep 09 04:28:06.615109 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.614983    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4" event={"ID":"247f310d-7fdf-47e5-a478-f8afeed670b7","Type":"ContainerDied","Data":"1df4bd2c96f9e7c60c4a408477ebd5d320b91be110c7227b48dc175d3bc507e4"}
Sep 09 04:28:06.615109 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.615017    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4" event={"ID":"247f310d-7fdf-47e5-a478-f8afeed670b7","Type":"ContainerDied","Data":"5de156357f87747a8fc01f3b7977b7b2ccacdf2b79fb73a26ecd2559d00cd14c"}
Sep 09 04:28:06.615109 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.615037    2553 scope.go:117] "RemoveContainer" containerID="1df4bd2c96f9e7c60c4a408477ebd5d320b91be110c7227b48dc175d3bc507e4"
Sep 09 04:28:06.625602 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.625578    2553 scope.go:117] "RemoveContainer" containerID="1df4bd2c96f9e7c60c4a408477ebd5d320b91be110c7227b48dc175d3bc507e4"
Sep 09 04:28:06.626020 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:28:06.625997    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df4bd2c96f9e7c60c4a408477ebd5d320b91be110c7227b48dc175d3bc507e4\": container with ID starting with 1df4bd2c96f9e7c60c4a408477ebd5d320b91be110c7227b48dc175d3bc507e4 not found: ID does not exist" containerID="1df4bd2c96f9e7c60c4a408477ebd5d320b91be110c7227b48dc175d3bc507e4"
Sep 09 04:28:06.626124 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.626032    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df4bd2c96f9e7c60c4a408477ebd5d320b91be110c7227b48dc175d3bc507e4"} err="failed to get container status \"1df4bd2c96f9e7c60c4a408477ebd5d320b91be110c7227b48dc175d3bc507e4\": rpc error: code = NotFound desc = could not find container \"1df4bd2c96f9e7c60c4a408477ebd5d320b91be110c7227b48dc175d3bc507e4\": container with ID starting with 1df4bd2c96f9e7c60c4a408477ebd5d320b91be110c7227b48dc175d3bc507e4 not found: ID does not exist"
Sep 09 04:28:06.636742 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.636720    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn4vq\" (UniqueName: \"kubernetes.io/projected/247f310d-7fdf-47e5-a478-f8afeed670b7-kube-api-access-hn4vq\") pod \"247f310d-7fdf-47e5-a478-f8afeed670b7\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") "
Sep 09 04:28:06.636838 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.636810    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247f310d-7fdf-47e5-a478-f8afeed670b7-tls-certs\") pod \"247f310d-7fdf-47e5-a478-f8afeed670b7\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") "
Sep 09 04:28:06.636893 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.636860    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-tmp-dir\") pod \"247f310d-7fdf-47e5-a478-f8afeed670b7\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") "
Sep 09 04:28:06.636893 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.636882    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-model-cache\") pod \"247f310d-7fdf-47e5-a478-f8afeed670b7\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") "
Sep 09 04:28:06.637006 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.636952    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-home\") pod \"247f310d-7fdf-47e5-a478-f8afeed670b7\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") "
Sep 09 04:28:06.637006 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.636971    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-dshm\") pod \"247f310d-7fdf-47e5-a478-f8afeed670b7\" (UID: \"247f310d-7fdf-47e5-a478-f8afeed670b7\") "
Sep 09 04:28:06.637179 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.637128    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "247f310d-7fdf-47e5-a478-f8afeed670b7" (UID: "247f310d-7fdf-47e5-a478-f8afeed670b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:28:06.637179 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.637144    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "247f310d-7fdf-47e5-a478-f8afeed670b7" (UID: "247f310d-7fdf-47e5-a478-f8afeed670b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:28:06.637404 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.637230    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-home" (OuterVolumeSpecName: "home") pod "247f310d-7fdf-47e5-a478-f8afeed670b7" (UID: "247f310d-7fdf-47e5-a478-f8afeed670b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:28:06.637404 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.637383    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:28:06.637404 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.637403    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:28:06.637556 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.637415    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:28:06.638895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.638874    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247f310d-7fdf-47e5-a478-f8afeed670b7-kube-api-access-hn4vq" (OuterVolumeSpecName: "kube-api-access-hn4vq") pod "247f310d-7fdf-47e5-a478-f8afeed670b7" (UID: "247f310d-7fdf-47e5-a478-f8afeed670b7"). InnerVolumeSpecName "kube-api-access-hn4vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:28:06.639187 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.639170    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247f310d-7fdf-47e5-a478-f8afeed670b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "247f310d-7fdf-47e5-a478-f8afeed670b7" (UID: "247f310d-7fdf-47e5-a478-f8afeed670b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:28:06.639282 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.639267    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-dshm" (OuterVolumeSpecName: "dshm") pod "247f310d-7fdf-47e5-a478-f8afeed670b7" (UID: "247f310d-7fdf-47e5-a478-f8afeed670b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:28:06.738430 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.738408    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247f310d-7fdf-47e5-a478-f8afeed670b7-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:28:06.738430 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.738427    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn4vq\" (UniqueName: \"kubernetes.io/projected/247f310d-7fdf-47e5-a478-f8afeed670b7-kube-api-access-hn4vq\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:28:06.738577 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.738438    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247f310d-7fdf-47e5-a478-f8afeed670b7-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:28:06.939929 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.939897    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"]
Sep 09 04:28:06.944792 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.944769    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9c44b5/rate-test-model-9c44b5-kserve-f66c4bcc9-hzps4"]
Sep 09 04:28:06.988878 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:06.988845    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247f310d-7fdf-47e5-a478-f8afeed670b7" path="/var/lib/kubelet/pods/247f310d-7fdf-47e5-a478-f8afeed670b7/volumes"
Sep 09 04:28:09.580762 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:09.580739    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"
Sep 09 04:28:20.539672 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:20.539638    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:21.233589 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:21.233554    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:21.539025 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:21.538933    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:21.753411 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:21.753355    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:24.250289 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:24.250245    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:24.365419 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:24.365385    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:24.454339 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:24.454310    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:25.602048 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:25.602014    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:25.702599 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:25.702570    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:26.157627 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.157600    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:26.257256 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.257231    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:26.273475 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.273452    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"]
Sep 09 04:28:26.273876 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.273865    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247f310d-7fdf-47e5-a478-f8afeed670b7" containerName="main"
Sep 09 04:28:26.273927 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.273878    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="247f310d-7fdf-47e5-a478-f8afeed670b7" containerName="main"
Sep 09 04:28:26.273962 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.273945    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="247f310d-7fdf-47e5-a478-f8afeed670b7" containerName="main"
Sep 09 04:28:26.276886 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.276871    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"
Sep 09 04:28:26.279038 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.279021    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0db87d\""
Sep 09 04:28:26.285593 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.285569    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"]
Sep 09 04:28:26.333538 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.333518    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:26.406842 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.406811    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pclp\" (UniqueName: \"kubernetes.io/projected/9eb2561a-0426-4a68-aa41-72ef8985b4e2-kube-api-access-5pclp\") pod \"payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8\" (UID: \"9eb2561a-0426-4a68-aa41-72ef8985b4e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"
Sep 09 04:28:26.407032 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.406895    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb2561a-0426-4a68-aa41-72ef8985b4e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8\" (UID: \"9eb2561a-0426-4a68-aa41-72ef8985b4e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"
Sep 09 04:28:26.508225 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.508203    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pclp\" (UniqueName: \"kubernetes.io/projected/9eb2561a-0426-4a68-aa41-72ef8985b4e2-kube-api-access-5pclp\") pod \"payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8\" (UID: \"9eb2561a-0426-4a68-aa41-72ef8985b4e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"
Sep 09 04:28:26.508388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.508240    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb2561a-0426-4a68-aa41-72ef8985b4e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8\" (UID: \"9eb2561a-0426-4a68-aa41-72ef8985b4e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"
Sep 09 04:28:26.508891 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.508872    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb2561a-0426-4a68-aa41-72ef8985b4e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8\" (UID: \"9eb2561a-0426-4a68-aa41-72ef8985b4e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"
Sep 09 04:28:26.526598 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.526575    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pclp\" (UniqueName: \"kubernetes.io/projected/9eb2561a-0426-4a68-aa41-72ef8985b4e2-kube-api-access-5pclp\") pod \"payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8\" (UID: \"9eb2561a-0426-4a68-aa41-72ef8985b4e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"
Sep 09 04:28:26.588515 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.588491    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"
Sep 09 04:28:26.859910 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:26.859887    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"]
Sep 09 04:28:26.862064 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:28:26.862031    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb2561a_0426_4a68_aa41_72ef8985b4e2.slice/crio-84a9dbdc338e81193d1881bdc23a1b61b55a96d99961bc4c2b435739b5313f66 WatchSource:0}: Error finding container 84a9dbdc338e81193d1881bdc23a1b61b55a96d99961bc4c2b435739b5313f66: Status 404 returned error can't find the container with id 84a9dbdc338e81193d1881bdc23a1b61b55a96d99961bc4c2b435739b5313f66
Sep 09 04:28:27.731032 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:27.730996    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8" event={"ID":"9eb2561a-0426-4a68-aa41-72ef8985b4e2","Type":"ContainerStarted","Data":"9874975cef52843681dc74c4d76da8e4d4eb3e50c0778a07c2fc274d23faac53"}
Sep 09 04:28:27.731209 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:27.731055    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8" event={"ID":"9eb2561a-0426-4a68-aa41-72ef8985b4e2","Type":"ContainerStarted","Data":"84a9dbdc338e81193d1881bdc23a1b61b55a96d99961bc4c2b435739b5313f66"}
Sep 09 04:28:27.731209 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:27.731081    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"
Sep 09 04:28:27.750414 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:27.750356    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8" podStartSLOduration=1.750341904 podStartE2EDuration="1.750341904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:28:27.749046019 +0000 UTC m=+1807.275416739" watchObservedRunningTime="2026-09-09 04:28:27.750341904 +0000 UTC m=+1807.276712613"
Sep 09 04:28:32.667944 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:32.667915    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:28:38.738470 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:38.738418    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"
Sep 09 04:28:39.237485 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:39.237438    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:44.879416 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:44.879386    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:46.569618 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:46.569576    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:49.356014 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:49.355972    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:59.663868 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.663827    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"]
Sep 09 04:28:59.667760 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.667741    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.670565 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.670546    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a3ce90\"/\"openshift-service-ca.crt\""
Sep 09 04:28:59.671650 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.671629    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a3ce90\"/\"default-dockercfg-fghbc\""
Sep 09 04:28:59.671650 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.671644    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a3ce90\"/\"ipp-route-a3ce90-kserve-self-signed-certs\""
Sep 09 04:28:59.671795 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.671653    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a3ce90\"/\"kube-root-ca.crt\""
Sep 09 04:28:59.678214 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.678170    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"]
Sep 09 04:28:59.685650 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.685628    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:28:59.799646 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.799611    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8v6r\" (UniqueName: \"kubernetes.io/projected/d3401d03-9be4-4483-bd6d-cdda7d171629-kube-api-access-s8v6r\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.799785 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.799696    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-dshm\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.799785 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.799728    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-tmp-dir\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.799785 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.799749    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-home\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.799929 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.799795    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3401d03-9be4-4483-bd6d-cdda7d171629-tls-certs\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.799929 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.799828    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-model-cache\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.900927 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.900902    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-tmp-dir\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.900927 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.900927    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-home\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.901112 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.900956    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3401d03-9be4-4483-bd6d-cdda7d171629-tls-certs\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.901112 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.900986    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-model-cache\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.901112 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.901071    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8v6r\" (UniqueName: \"kubernetes.io/projected/d3401d03-9be4-4483-bd6d-cdda7d171629-kube-api-access-s8v6r\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.901289 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.901116    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-dshm\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.901350 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.901298    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-tmp-dir\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.901401 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.901340    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-home\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.901468 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.901396    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-model-cache\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.903407 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.903382    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-dshm\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.903587 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.903570    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3401d03-9be4-4483-bd6d-cdda7d171629-tls-certs\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.909916 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.909897    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8v6r\" (UniqueName: \"kubernetes.io/projected/d3401d03-9be4-4483-bd6d-cdda7d171629-kube-api-access-s8v6r\") pod \"ipp-route-a3ce90-kserve-59fdcd9995-v6hcc\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") " pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:28:59.979063 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:28:59.979038    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:29:00.253872 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:00.253849    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"]
Sep 09 04:29:00.255511 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:29:00.255477    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3401d03_9be4_4483_bd6d_cdda7d171629.slice/crio-88544e1f75e749ca441232f5ff0948055665288d4096b1a92e851ddbcdb53fc0 WatchSource:0}: Error finding container 88544e1f75e749ca441232f5ff0948055665288d4096b1a92e851ddbcdb53fc0: Status 404 returned error can't find the container with id 88544e1f75e749ca441232f5ff0948055665288d4096b1a92e851ddbcdb53fc0
Sep 09 04:29:00.907075 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:00.907042    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc" event={"ID":"d3401d03-9be4-4483-bd6d-cdda7d171629","Type":"ContainerStarted","Data":"705f8529cfe149bd90cc6bf6e2377e3b777f61de1fcfc8ace82aa934f5759e25"}
Sep 09 04:29:00.907585 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:00.907089    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc" event={"ID":"d3401d03-9be4-4483-bd6d-cdda7d171629","Type":"ContainerStarted","Data":"88544e1f75e749ca441232f5ff0948055665288d4096b1a92e851ddbcdb53fc0"}
Sep 09 04:29:00.926322 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:00.926274    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc" podStartSLOduration=1.926258871 podStartE2EDuration="1.926258871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:29:00.924570391 +0000 UTC m=+1840.450941113" watchObservedRunningTime="2026-09-09 04:29:00.926258871 +0000 UTC m=+1840.452629632"
Sep 09 04:29:03.769624 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:03.769590    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:29:09.979989 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:09.979960    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:29:09.980465 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:09.980008    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:29:09.992086 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:09.992066    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:29:10.970425 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:10.970389    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:29:13.386367 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:13.386334    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:29:21.376627 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:21.376597    2553 scope.go:117] "RemoveContainer" containerID="a9ca130e43501e1c3540d0d76b4a3c6a0dcb668ce467cd05e447a9355d27d60c"
Sep 09 04:29:21.385679 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:21.384980    2553 scope.go:117] "RemoveContainer" containerID="d98495cadf5b7beb2f0b892f7355d04d084509608fc69fe605d8509656265390"
Sep 09 04:29:24.269490 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:24.269455    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:29:26.781195 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:26.781167    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:29:30.342809 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:30.342776    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz_5b5d4fd5-d238-4dbd-8526-6a0c95762f39/payload-processing/0.log"
Sep 09 04:29:30.872754 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:30.872719    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:29:31.179405 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:31.179331    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:29:31.233071 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:31.233043    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:29:34.078047 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:34.077998    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:29:34.170128 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:34.170094    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:29:35.414958 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:35.414921    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:29:35.516278 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:35.516244    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:29:35.615400 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:35.615371    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:29:35.915723 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:35.915693    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:29:36.015723 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:36.015687    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:29:36.058024 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:36.057999    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"]
Sep 09 04:29:36.061911 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:36.061896    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"
Sep 09 04:29:36.064272 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:36.064252    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-e787bf\""
Sep 09 04:29:36.064389 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:36.064254    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-e787bf-dockercfg-d8d6s\""
Sep 09 04:29:36.072714 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:36.072693    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"]
Sep 09 04:29:36.124552 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:36.124512    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cbe161-c80c-47ac-85c4-1b54f94f2d62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq\" (UID: \"b0cbe161-c80c-47ac-85c4-1b54f94f2d62\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"
Sep 09 04:29:36.124696 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:36.124623    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56w5x\" (UniqueName: \"kubernetes.io/projected/b0cbe161-c80c-47ac-85c4-1b54f94f2d62-kube-api-access-56w5x\") pod \"payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq\" (UID: \"b0cbe161-c80c-47ac-85c4-1b54f94f2d62\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"
Sep 09 04:29:36.225557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:36.225522    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cbe161-c80c-47ac-85c4-1b54f94f2d62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq\" (UID: \"b0cbe161-c80c-47ac-85c4-1b54f94f2d62\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"
Sep 09 04:29:36.225702 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:36.225607    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56w5x\" (UniqueName: \"kubernetes.io/projected/b0cbe161-c80c-47ac-85c4-1b54f94f2d62-kube-api-access-56w5x\") pod \"payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq\" (UID: \"b0cbe161-c80c-47ac-85c4-1b54f94f2d62\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"
Sep 09 04:29:36.226258 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:36.226239    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cbe161-c80c-47ac-85c4-1b54f94f2d62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq\" (UID: \"b0cbe161-c80c-47ac-85c4-1b54f94f2d62\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"
Sep 09 04:29:36.247639 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:36.247617    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56w5x\" (UniqueName: \"kubernetes.io/projected/b0cbe161-c80c-47ac-85c4-1b54f94f2d62-kube-api-access-56w5x\") pod \"payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq\" (UID: \"b0cbe161-c80c-47ac-85c4-1b54f94f2d62\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"
Sep 09 04:29:36.373641 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:36.373613    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"
Sep 09 04:29:36.644837 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:36.644814    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"]
Sep 09 04:29:36.646638 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:29:36.646612    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cbe161_c80c_47ac_85c4_1b54f94f2d62.slice/crio-e0083c6fa5280d5046c93a22f1f9481aa697a844b98e799320d84686f30ed342 WatchSource:0}: Error finding container e0083c6fa5280d5046c93a22f1f9481aa697a844b98e799320d84686f30ed342: Status 404 returned error can't find the container with id e0083c6fa5280d5046c93a22f1f9481aa697a844b98e799320d84686f30ed342
Sep 09 04:29:37.101833 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:37.101800    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq" event={"ID":"b0cbe161-c80c-47ac-85c4-1b54f94f2d62","Type":"ContainerStarted","Data":"e7bedfce40236f236164b30298e19ee42e38ffbd8ed96400792579e1912a656b"}
Sep 09 04:29:37.102030 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:37.101850    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq" event={"ID":"b0cbe161-c80c-47ac-85c4-1b54f94f2d62","Type":"ContainerStarted","Data":"e0083c6fa5280d5046c93a22f1f9481aa697a844b98e799320d84686f30ed342"}
Sep 09 04:29:37.102030 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:37.101907    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"
Sep 09 04:29:37.120399 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:37.120361    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq" podStartSLOduration=1.120349101 podStartE2EDuration="1.120349101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:29:37.119829896 +0000 UTC m=+1876.646200606" watchObservedRunningTime="2026-09-09 04:29:37.120349101 +0000 UTC m=+1876.646719810"
Sep 09 04:29:42.779001 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:42.778929    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:29:48.109362 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:48.109327    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"
Sep 09 04:29:59.068600 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:59.068541    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:29:59.772540 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:59.772507    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75"]
Sep 09 04:29:59.776169 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:59.776153    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75"
Sep 09 04:29:59.778607 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:59.778585    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:29:59.778724 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:59.778616    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:29:59.784246 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:59.784223    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75"]
Sep 09 04:29:59.928696 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:29:59.928669    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42094478-5010-4c81-ad76-9c8f110a111c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75\" (UID: \"42094478-5010-4c81-ad76-9c8f110a111c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75"
Sep 09 04:30:00.030008 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.029940    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42094478-5010-4c81-ad76-9c8f110a111c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75\" (UID: \"42094478-5010-4c81-ad76-9c8f110a111c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75"
Sep 09 04:30:00.030490 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.030474    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42094478-5010-4c81-ad76-9c8f110a111c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75\" (UID: \"42094478-5010-4c81-ad76-9c8f110a111c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75"
Sep 09 04:30:00.087391 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.087372    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75"
Sep 09 04:30:00.139361 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.139318    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9c44b5-29815470-xhdzd"]
Sep 09 04:30:00.148823 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.148784    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815470-kf77j"]
Sep 09 04:30:00.148946 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.148920    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9c44b5-29815470-xhdzd"
Sep 09 04:30:00.155136 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.153083    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0db87d-29815470-m6dzs"]
Sep 09 04:30:00.155136 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.154552    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815470-kf77j"
Sep 09 04:30:00.167538 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.167267    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9c44b5-29815470-xhdzd"]
Sep 09 04:30:00.167538 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.167311    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0db87d-29815470-m6dzs"]
Sep 09 04:30:00.167538 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.167392    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0db87d-29815470-m6dzs"
Sep 09 04:30:00.173348 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.171950    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9c44b5-29815470-xhdzd"
Sep 09 04:30:00.173557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.173540    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815470-kf77j"]
Sep 09 04:30:00.179391 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.179324    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-e787bf-29815470-wm6fr"]
Sep 09 04:30:00.185338 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.185171    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815470-kf77j"
Sep 09 04:30:00.189932 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.189669    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-e787bf-29815470-wm6fr"]
Sep 09 04:30:00.189932 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.189762    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-e787bf-29815470-wm6fr"
Sep 09 04:30:00.216247 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.216073    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0db87d-29815470-m6dzs"
Sep 09 04:30:00.228605 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.228587    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-e787bf-29815470-wm6fr"
Sep 09 04:30:00.366872 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.366848    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75"]
Sep 09 04:30:00.369335 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:30:00.369285    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42094478_5010_4c81_ad76_9c8f110a111c.slice/crio-3964807f7c6762c806a87075f834299331bf7b277d02df1dad03de3406801f90 WatchSource:0}: Error finding container 3964807f7c6762c806a87075f834299331bf7b277d02df1dad03de3406801f90: Status 404 returned error can't find the container with id 3964807f7c6762c806a87075f834299331bf7b277d02df1dad03de3406801f90
Sep 09 04:30:00.471336 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.471307    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9c44b5-29815470-xhdzd"]
Sep 09 04:30:00.474605 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:30:00.474580    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e365790_553d_4d9d_9a8f_59eaf55c7524.slice/crio-d16ed15688960e44d7a74e16e58e7c4da5dd407c5c6eb7de1d000c0b58712eeb WatchSource:0}: Error finding container d16ed15688960e44d7a74e16e58e7c4da5dd407c5c6eb7de1d000c0b58712eeb: Status 404 returned error can't find the container with id d16ed15688960e44d7a74e16e58e7c4da5dd407c5c6eb7de1d000c0b58712eeb
Sep 09 04:30:00.475270 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.475227    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815470-kf77j"]
Sep 09 04:30:00.475865 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:30:00.475845    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7150de37_2f88_4075_9104_048a537a21e2.slice/crio-518238c9013c879dae057bf618f4287896574647dfa6dc5f90d16cd56cdb794e WatchSource:0}: Error finding container 518238c9013c879dae057bf618f4287896574647dfa6dc5f90d16cd56cdb794e: Status 404 returned error can't find the container with id 518238c9013c879dae057bf618f4287896574647dfa6dc5f90d16cd56cdb794e
Sep 09 04:30:00.515260 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.515209    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0db87d-29815470-m6dzs"]
Sep 09 04:30:00.517120 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:30:00.517098    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44b1cb2_c8e5_4ce5_9b7a_87c7b23b44f9.slice/crio-3d37b9209b7866fb930fc24ed55815139c81f7b2ab6d5e3d51ef109376183576 WatchSource:0}: Error finding container 3d37b9209b7866fb930fc24ed55815139c81f7b2ab6d5e3d51ef109376183576: Status 404 returned error can't find the container with id 3d37b9209b7866fb930fc24ed55815139c81f7b2ab6d5e3d51ef109376183576
Sep 09 04:30:00.528924 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:00.528903    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-e787bf-29815470-wm6fr"]
Sep 09 04:30:00.531004 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:30:00.530984    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958c407a_bbd5_4c98_b048_285adadd4df4.slice/crio-5f2a040d2fdee1f5e165ee7f05e3ffb835d46c8cef073efb3b0cfac226b390ee WatchSource:0}: Error finding container 5f2a040d2fdee1f5e165ee7f05e3ffb835d46c8cef073efb3b0cfac226b390ee: Status 404 returned error can't find the container with id 5f2a040d2fdee1f5e165ee7f05e3ffb835d46c8cef073efb3b0cfac226b390ee
Sep 09 04:30:01.233024 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.232990    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815470-kf77j" event={"ID":"7150de37-2f88-4075-9104-048a537a21e2","Type":"ContainerStarted","Data":"44464db977dca25e602bd000fe9efc60b22c322c963b1b69d53f3a096b95db8b"}
Sep 09 04:30:01.233383 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.233055    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815470-kf77j" event={"ID":"7150de37-2f88-4075-9104-048a537a21e2","Type":"ContainerStarted","Data":"518238c9013c879dae057bf618f4287896574647dfa6dc5f90d16cd56cdb794e"}
Sep 09 04:30:01.234599 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.234573    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9c44b5-29815470-xhdzd" event={"ID":"2e365790-553d-4d9d-9a8f-59eaf55c7524","Type":"ContainerStarted","Data":"38310d4dbbad9004d8d1b99c6d7fcbe0b61149b42a2bec2853121487f3a302a7"}
Sep 09 04:30:01.234713 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.234627    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9c44b5-29815470-xhdzd" event={"ID":"2e365790-553d-4d9d-9a8f-59eaf55c7524","Type":"ContainerStarted","Data":"d16ed15688960e44d7a74e16e58e7c4da5dd407c5c6eb7de1d000c0b58712eeb"}
Sep 09 04:30:01.235948 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.235928    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0db87d-29815470-m6dzs" event={"ID":"e44b1cb2-c8e5-4ce5-9b7a-87c7b23b44f9","Type":"ContainerStarted","Data":"a93fc2d93373799f61d06c1a9c87b695f87cca5947b08779fc6a46b1064bbd8e"}
Sep 09 04:30:01.236012 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.235968    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0db87d-29815470-m6dzs" event={"ID":"e44b1cb2-c8e5-4ce5-9b7a-87c7b23b44f9","Type":"ContainerStarted","Data":"3d37b9209b7866fb930fc24ed55815139c81f7b2ab6d5e3d51ef109376183576"}
Sep 09 04:30:01.237510 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.237481    2553 generic.go:356] "Generic (PLEG): container finished" podID="42094478-5010-4c81-ad76-9c8f110a111c" containerID="5abac9166743b7559aabede0050bf51f57748cca2ef686af7bdbb318abf8b6a9" exitCode=0
Sep 09 04:30:01.237599 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.237520    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75" event={"ID":"42094478-5010-4c81-ad76-9c8f110a111c","Type":"ContainerDied","Data":"5abac9166743b7559aabede0050bf51f57748cca2ef686af7bdbb318abf8b6a9"}
Sep 09 04:30:01.237599 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.237562    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75" event={"ID":"42094478-5010-4c81-ad76-9c8f110a111c","Type":"ContainerStarted","Data":"3964807f7c6762c806a87075f834299331bf7b277d02df1dad03de3406801f90"}
Sep 09 04:30:01.238903 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.238883    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-e787bf-29815470-wm6fr" event={"ID":"958c407a-bbd5-4c98-b048-285adadd4df4","Type":"ContainerStarted","Data":"3cfcdc1ae943d8893a27bfb6fbbeb4e840869234d672853bbf1066a8029b7af0"}
Sep 09 04:30:01.238978 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.238918    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-e787bf-29815470-wm6fr" event={"ID":"958c407a-bbd5-4c98-b048-285adadd4df4","Type":"ContainerStarted","Data":"5f2a040d2fdee1f5e165ee7f05e3ffb835d46c8cef073efb3b0cfac226b390ee"}
Sep 09 04:30:01.266436 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.266377    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0db87d-29815470-m6dzs" podStartSLOduration=1.2663643119999999 podStartE2EDuration="1.266364312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:30:01.265987762 +0000 UTC m=+1900.792358473" watchObservedRunningTime="2026-09-09 04:30:01.266364312 +0000 UTC m=+1900.792735022"
Sep 09 04:30:01.267632 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.267599    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815470-kf77j" podStartSLOduration=1.267587603 podStartE2EDuration="1.267587603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:30:01.24973991 +0000 UTC m=+1900.776110622" watchObservedRunningTime="2026-09-09 04:30:01.267587603 +0000 UTC m=+1900.793958306"
Sep 09 04:30:01.281110 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.281075    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-e787bf-29815470-wm6fr" podStartSLOduration=1.281062676 podStartE2EDuration="1.281062676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:30:01.280438023 +0000 UTC m=+1900.806808732" watchObservedRunningTime="2026-09-09 04:30:01.281062676 +0000 UTC m=+1900.807433384"
Sep 09 04:30:01.297998 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:01.297952    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9c44b5-29815470-xhdzd" podStartSLOduration=1.297934631 podStartE2EDuration="1.297934631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:30:01.295508796 +0000 UTC m=+1900.821879506" watchObservedRunningTime="2026-09-09 04:30:01.297934631 +0000 UTC m=+1900.824305343"
Sep 09 04:30:02.245814 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:02.245780    2553 generic.go:356] "Generic (PLEG): container finished" podID="e44b1cb2-c8e5-4ce5-9b7a-87c7b23b44f9" containerID="a93fc2d93373799f61d06c1a9c87b695f87cca5947b08779fc6a46b1064bbd8e" exitCode=0
Sep 09 04:30:02.246247 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:02.245824    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0db87d-29815470-m6dzs" event={"ID":"e44b1cb2-c8e5-4ce5-9b7a-87c7b23b44f9","Type":"ContainerDied","Data":"a93fc2d93373799f61d06c1a9c87b695f87cca5947b08779fc6a46b1064bbd8e"}
Sep 09 04:30:02.251355 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:02.251333    2553 generic.go:356] "Generic (PLEG): container finished" podID="958c407a-bbd5-4c98-b048-285adadd4df4" containerID="3cfcdc1ae943d8893a27bfb6fbbeb4e840869234d672853bbf1066a8029b7af0" exitCode=0
Sep 09 04:30:02.251497 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:02.251399    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-e787bf-29815470-wm6fr" event={"ID":"958c407a-bbd5-4c98-b048-285adadd4df4","Type":"ContainerDied","Data":"3cfcdc1ae943d8893a27bfb6fbbeb4e840869234d672853bbf1066a8029b7af0"}
Sep 09 04:30:02.252969 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:02.252945    2553 generic.go:356] "Generic (PLEG): container finished" podID="7150de37-2f88-4075-9104-048a537a21e2" containerID="44464db977dca25e602bd000fe9efc60b22c322c963b1b69d53f3a096b95db8b" exitCode=0
Sep 09 04:30:02.253098 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:02.253022    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815470-kf77j" event={"ID":"7150de37-2f88-4075-9104-048a537a21e2","Type":"ContainerDied","Data":"44464db977dca25e602bd000fe9efc60b22c322c963b1b69d53f3a096b95db8b"}
Sep 09 04:30:02.254505 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:02.254486    2553 generic.go:356] "Generic (PLEG): container finished" podID="2e365790-553d-4d9d-9a8f-59eaf55c7524" containerID="38310d4dbbad9004d8d1b99c6d7fcbe0b61149b42a2bec2853121487f3a302a7" exitCode=0
Sep 09 04:30:02.254605 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:02.254534    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9c44b5-29815470-xhdzd" event={"ID":"2e365790-553d-4d9d-9a8f-59eaf55c7524","Type":"ContainerDied","Data":"38310d4dbbad9004d8d1b99c6d7fcbe0b61149b42a2bec2853121487f3a302a7"}
Sep 09 04:30:02.396165 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:02.396140    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75"
Sep 09 04:30:02.553869 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:02.553838    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42094478-5010-4c81-ad76-9c8f110a111c-maas-api-service-ca\") pod \"42094478-5010-4c81-ad76-9c8f110a111c\" (UID: \"42094478-5010-4c81-ad76-9c8f110a111c\") "
Sep 09 04:30:02.554228 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:02.554207    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42094478-5010-4c81-ad76-9c8f110a111c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42094478-5010-4c81-ad76-9c8f110a111c" (UID: "42094478-5010-4c81-ad76-9c8f110a111c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:30:02.655159 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:02.655095    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42094478-5010-4c81-ad76-9c8f110a111c-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:30:03.261796 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:03.261725    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75"
Sep 09 04:30:03.261796 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:03.261735    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75" event={"ID":"42094478-5010-4c81-ad76-9c8f110a111c","Type":"ContainerDied","Data":"3964807f7c6762c806a87075f834299331bf7b277d02df1dad03de3406801f90"}
Sep 09 04:30:03.261796 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:03.261777    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3964807f7c6762c806a87075f834299331bf7b277d02df1dad03de3406801f90"
Sep 09 04:30:03.416680 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:03.414989    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9c44b5-29815470-xhdzd"
Sep 09 04:30:03.449020 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:03.448995    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815470-kf77j"
Sep 09 04:30:03.478302 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:03.478278    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0db87d-29815470-m6dzs"
Sep 09 04:30:03.485356 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:03.483655    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-e787bf-29815470-wm6fr"
Sep 09 04:30:04.268956 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:04.268927    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815470-kf77j"
Sep 09 04:30:04.269388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:04.268957    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815470-kf77j" event={"ID":"7150de37-2f88-4075-9104-048a537a21e2","Type":"ContainerDied","Data":"518238c9013c879dae057bf618f4287896574647dfa6dc5f90d16cd56cdb794e"}
Sep 09 04:30:04.269388 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:04.269020    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="518238c9013c879dae057bf618f4287896574647dfa6dc5f90d16cd56cdb794e"
Sep 09 04:30:04.270307 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:04.270278    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9c44b5-29815470-xhdzd"
Sep 09 04:30:04.270432 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:04.270308    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9c44b5-29815470-xhdzd" event={"ID":"2e365790-553d-4d9d-9a8f-59eaf55c7524","Type":"ContainerDied","Data":"d16ed15688960e44d7a74e16e58e7c4da5dd407c5c6eb7de1d000c0b58712eeb"}
Sep 09 04:30:04.270432 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:04.270345    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d16ed15688960e44d7a74e16e58e7c4da5dd407c5c6eb7de1d000c0b58712eeb"
Sep 09 04:30:04.271829 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:04.271812    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0db87d-29815470-m6dzs"
Sep 09 04:30:04.271935 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:04.271853    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0db87d-29815470-m6dzs" event={"ID":"e44b1cb2-c8e5-4ce5-9b7a-87c7b23b44f9","Type":"ContainerDied","Data":"3d37b9209b7866fb930fc24ed55815139c81f7b2ab6d5e3d51ef109376183576"}
Sep 09 04:30:04.271935 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:04.271892    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d37b9209b7866fb930fc24ed55815139c81f7b2ab6d5e3d51ef109376183576"
Sep 09 04:30:04.273330 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:04.273307    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-e787bf-29815470-wm6fr" event={"ID":"958c407a-bbd5-4c98-b048-285adadd4df4","Type":"ContainerDied","Data":"5f2a040d2fdee1f5e165ee7f05e3ffb835d46c8cef073efb3b0cfac226b390ee"}
Sep 09 04:30:04.273330 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:04.273325    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-e787bf-29815470-wm6fr"
Sep 09 04:30:04.273526 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:04.273348    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2a040d2fdee1f5e165ee7f05e3ffb835d46c8cef073efb3b0cfac226b390ee"
Sep 09 04:30:06.019707 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.019661    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:30:06.031262 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.031233    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-e787bf-29815470-wm6fr"]
Sep 09 04:30:06.034760 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.034737    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-e787bf-29815470-wm6fr"]
Sep 09 04:30:06.059769 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.059739    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"]
Sep 09 04:30:06.060025 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.059992    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq" podUID="b0cbe161-c80c-47ac-85c4-1b54f94f2d62" containerName="payload-processing" containerID="cri-o://e7bedfce40236f236164b30298e19ee42e38ffbd8ed96400792579e1912a656b" gracePeriod=30
Sep 09 04:30:06.119704 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.119668    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:30:06.251585 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.251393    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75"]
Sep 09 04:30:06.254398 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.254363    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e787bf-hgp75"]
Sep 09 04:30:06.287317 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.287246    2553 generic.go:356] "Generic (PLEG): container finished" podID="b0cbe161-c80c-47ac-85c4-1b54f94f2d62" containerID="e7bedfce40236f236164b30298e19ee42e38ffbd8ed96400792579e1912a656b" exitCode=0
Sep 09 04:30:06.287507 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.287313    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq" event={"ID":"b0cbe161-c80c-47ac-85c4-1b54f94f2d62","Type":"ContainerDied","Data":"e7bedfce40236f236164b30298e19ee42e38ffbd8ed96400792579e1912a656b"}
Sep 09 04:30:06.313876 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.313852    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"
Sep 09 04:30:06.385843 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.385817    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56w5x\" (UniqueName: \"kubernetes.io/projected/b0cbe161-c80c-47ac-85c4-1b54f94f2d62-kube-api-access-56w5x\") pod \"b0cbe161-c80c-47ac-85c4-1b54f94f2d62\" (UID: \"b0cbe161-c80c-47ac-85c4-1b54f94f2d62\") "
Sep 09 04:30:06.385964 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.385867    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cbe161-c80c-47ac-85c4-1b54f94f2d62-plugins-config-volume\") pod \"b0cbe161-c80c-47ac-85c4-1b54f94f2d62\" (UID: \"b0cbe161-c80c-47ac-85c4-1b54f94f2d62\") "
Sep 09 04:30:06.386228 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.386208    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cbe161-c80c-47ac-85c4-1b54f94f2d62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b0cbe161-c80c-47ac-85c4-1b54f94f2d62" (UID: "b0cbe161-c80c-47ac-85c4-1b54f94f2d62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:30:06.388031 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.387986    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cbe161-c80c-47ac-85c4-1b54f94f2d62-kube-api-access-56w5x" (OuterVolumeSpecName: "kube-api-access-56w5x") pod "b0cbe161-c80c-47ac-85c4-1b54f94f2d62" (UID: "b0cbe161-c80c-47ac-85c4-1b54f94f2d62"). InnerVolumeSpecName "kube-api-access-56w5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:30:06.426773 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.426737    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:30:06.484463 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.484416    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:30:06.487154 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.487126    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56w5x\" (UniqueName: \"kubernetes.io/projected/b0cbe161-c80c-47ac-85c4-1b54f94f2d62-kube-api-access-56w5x\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:30:06.487154 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.487151    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cbe161-c80c-47ac-85c4-1b54f94f2d62-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:30:06.626228 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.626149    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:30:06.989405 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.989377    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42094478-5010-4c81-ad76-9c8f110a111c" path="/var/lib/kubelet/pods/42094478-5010-4c81-ad76-9c8f110a111c/volumes"
Sep 09 04:30:06.995483 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:06.990214    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="958c407a-bbd5-4c98-b048-285adadd4df4" path="/var/lib/kubelet/pods/958c407a-bbd5-4c98-b048-285adadd4df4/volumes"
Sep 09 04:30:07.293731 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:07.293657    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq" event={"ID":"b0cbe161-c80c-47ac-85c4-1b54f94f2d62","Type":"ContainerDied","Data":"e0083c6fa5280d5046c93a22f1f9481aa697a844b98e799320d84686f30ed342"}
Sep 09 04:30:07.293731 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:07.293704    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"
Sep 09 04:30:07.293731 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:07.293715    2553 scope.go:117] "RemoveContainer" containerID="e7bedfce40236f236164b30298e19ee42e38ffbd8ed96400792579e1912a656b"
Sep 09 04:30:07.317112 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:07.317085    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"]
Sep 09 04:30:07.321539 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:07.321519    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e787bf-85bddcd8f-jmrdq"]
Sep 09 04:30:08.989221 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:08.989191    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0cbe161-c80c-47ac-85c4-1b54f94f2d62" path="/var/lib/kubelet/pods/b0cbe161-c80c-47ac-85c4-1b54f94f2d62/volumes"
Sep 09 04:30:11.163601 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:11.163565    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:30:12.920519 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:12.920474    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:30:13.021618 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:13.021577    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:30:13.240014 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:13.239983    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:30:16.798906 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.798873    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc"]
Sep 09 04:30:16.799295 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799283    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44b1cb2-c8e5-4ce5-9b7a-87c7b23b44f9" containerName="cleanup"
Sep 09 04:30:16.799295 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799296    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44b1cb2-c8e5-4ce5-9b7a-87c7b23b44f9" containerName="cleanup"
Sep 09 04:30:16.799373 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799314    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7150de37-2f88-4075-9104-048a537a21e2" containerName="cleanup"
Sep 09 04:30:16.799373 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799319    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7150de37-2f88-4075-9104-048a537a21e2" containerName="cleanup"
Sep 09 04:30:16.799373 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799330    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42094478-5010-4c81-ad76-9c8f110a111c" containerName="revoke-keys"
Sep 09 04:30:16.799373 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799335    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="42094478-5010-4c81-ad76-9c8f110a111c" containerName="revoke-keys"
Sep 09 04:30:16.799373 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799341    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="958c407a-bbd5-4c98-b048-285adadd4df4" containerName="cleanup"
Sep 09 04:30:16.799373 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799346    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="958c407a-bbd5-4c98-b048-285adadd4df4" containerName="cleanup"
Sep 09 04:30:16.799373 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799352    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0cbe161-c80c-47ac-85c4-1b54f94f2d62" containerName="payload-processing"
Sep 09 04:30:16.799373 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799358    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0cbe161-c80c-47ac-85c4-1b54f94f2d62" containerName="payload-processing"
Sep 09 04:30:16.799373 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799368    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e365790-553d-4d9d-9a8f-59eaf55c7524" containerName="cleanup"
Sep 09 04:30:16.799373 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799373    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e365790-553d-4d9d-9a8f-59eaf55c7524" containerName="cleanup"
Sep 09 04:30:16.799699 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799454    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="42094478-5010-4c81-ad76-9c8f110a111c" containerName="revoke-keys"
Sep 09 04:30:16.799699 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799465    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e365790-553d-4d9d-9a8f-59eaf55c7524" containerName="cleanup"
Sep 09 04:30:16.799699 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799473    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0cbe161-c80c-47ac-85c4-1b54f94f2d62" containerName="payload-processing"
Sep 09 04:30:16.799699 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799477    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e44b1cb2-c8e5-4ce5-9b7a-87c7b23b44f9" containerName="cleanup"
Sep 09 04:30:16.799699 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799484    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7150de37-2f88-4075-9104-048a537a21e2" containerName="cleanup"
Sep 09 04:30:16.799699 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.799489    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="958c407a-bbd5-4c98-b048-285adadd4df4" containerName="cleanup"
Sep 09 04:30:16.803854 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.803835    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc"
Sep 09 04:30:16.806293 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.806275    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:30:16.806486 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.806469    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:30:16.810936 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.810913    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc"]
Sep 09 04:30:16.906071 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.906016    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:30:16.982918 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:16.982886    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af900ad3-aed8-4f48-b718-b70ee90c1972-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc\" (UID: \"af900ad3-aed8-4f48-b718-b70ee90c1972\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc"
Sep 09 04:30:17.084476 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:17.084380    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af900ad3-aed8-4f48-b718-b70ee90c1972-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc\" (UID: \"af900ad3-aed8-4f48-b718-b70ee90c1972\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc"
Sep 09 04:30:17.084991 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:17.084971    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af900ad3-aed8-4f48-b718-b70ee90c1972-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc\" (UID: \"af900ad3-aed8-4f48-b718-b70ee90c1972\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc"
Sep 09 04:30:17.116335 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:17.116307    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc"
Sep 09 04:30:17.390917 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:17.390894    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc"]
Sep 09 04:30:17.392533 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:30:17.392508    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf900ad3_aed8_4f48_b718_b70ee90c1972.slice/crio-e7e06ca150030002d8af25f679a242a4f97420278a05f2e901e5fff49138ba48 WatchSource:0}: Error finding container e7e06ca150030002d8af25f679a242a4f97420278a05f2e901e5fff49138ba48: Status 404 returned error can't find the container with id e7e06ca150030002d8af25f679a242a4f97420278a05f2e901e5fff49138ba48
Sep 09 04:30:18.355837 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:18.355764    2553 generic.go:356] "Generic (PLEG): container finished" podID="af900ad3-aed8-4f48-b718-b70ee90c1972" containerID="1b4bb9c14f541aec86d28de774132f4d77e083ffcf0109d771ef9d34047da53d" exitCode=0
Sep 09 04:30:18.356176 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:18.355848    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc" event={"ID":"af900ad3-aed8-4f48-b718-b70ee90c1972","Type":"ContainerDied","Data":"1b4bb9c14f541aec86d28de774132f4d77e083ffcf0109d771ef9d34047da53d"}
Sep 09 04:30:18.356176 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:18.355895    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc" event={"ID":"af900ad3-aed8-4f48-b718-b70ee90c1972","Type":"ContainerStarted","Data":"e7e06ca150030002d8af25f679a242a4f97420278a05f2e901e5fff49138ba48"}
Sep 09 04:30:19.493713 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:19.493686    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc"
Sep 09 04:30:19.506788 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:19.506763    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af900ad3-aed8-4f48-b718-b70ee90c1972-maas-api-service-ca\") pod \"af900ad3-aed8-4f48-b718-b70ee90c1972\" (UID: \"af900ad3-aed8-4f48-b718-b70ee90c1972\") "
Sep 09 04:30:19.507089 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:19.507068    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af900ad3-aed8-4f48-b718-b70ee90c1972-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af900ad3-aed8-4f48-b718-b70ee90c1972" (UID: "af900ad3-aed8-4f48-b718-b70ee90c1972"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:30:19.608383 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:19.608358    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af900ad3-aed8-4f48-b718-b70ee90c1972-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:30:20.368574 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:20.368548    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc"
Sep 09 04:30:20.368739 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:20.368586    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc" event={"ID":"af900ad3-aed8-4f48-b718-b70ee90c1972","Type":"ContainerDied","Data":"e7e06ca150030002d8af25f679a242a4f97420278a05f2e901e5fff49138ba48"}
Sep 09 04:30:20.368739 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:20.368625    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7e06ca150030002d8af25f679a242a4f97420278a05f2e901e5fff49138ba48"
Sep 09 04:30:21.408804 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:21.408772    2553 scope.go:117] "RemoveContainer" containerID="db536a41553cac559ace08e6f7a85e206cb98c287e255bfb3d1b5aec20acf127"
Sep 09 04:30:21.417176 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:21.417158    2553 scope.go:117] "RemoveContainer" containerID="c8e4d2327a27622b9258963ebd379a68fb5fe0be1372e2f80dadbbd5428e15eb"
Sep 09 04:30:30.758021 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:30.757992    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:30:32.540309 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:32.540272    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"]
Sep 09 04:30:32.540867 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:32.540558    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz" podUID="5b5d4fd5-d238-4dbd-8526-6a0c95762f39" containerName="payload-processing" containerID="cri-o://9e6c7174106c5c377cc28283c05ba278d3acd83cfe7d2d86f23de04135d136ca" gracePeriod=30
Sep 09 04:30:32.734613 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:32.734584    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc"]
Sep 09 04:30:32.738197 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:32.738169    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ce90-6c8gc"]
Sep 09 04:30:32.792512 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:32.792437    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"
Sep 09 04:30:32.829053 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:32.829026    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvgqq\" (UniqueName: \"kubernetes.io/projected/5b5d4fd5-d238-4dbd-8526-6a0c95762f39-kube-api-access-xvgqq\") pod \"5b5d4fd5-d238-4dbd-8526-6a0c95762f39\" (UID: \"5b5d4fd5-d238-4dbd-8526-6a0c95762f39\") "
Sep 09 04:30:32.829221 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:32.829176    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5d4fd5-d238-4dbd-8526-6a0c95762f39-plugins-config-volume\") pod \"5b5d4fd5-d238-4dbd-8526-6a0c95762f39\" (UID: \"5b5d4fd5-d238-4dbd-8526-6a0c95762f39\") "
Sep 09 04:30:32.829556 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:32.829534    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5d4fd5-d238-4dbd-8526-6a0c95762f39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b5d4fd5-d238-4dbd-8526-6a0c95762f39" (UID: "5b5d4fd5-d238-4dbd-8526-6a0c95762f39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:30:32.831233 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:32.831213    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5d4fd5-d238-4dbd-8526-6a0c95762f39-kube-api-access-xvgqq" (OuterVolumeSpecName: "kube-api-access-xvgqq") pod "5b5d4fd5-d238-4dbd-8526-6a0c95762f39" (UID: "5b5d4fd5-d238-4dbd-8526-6a0c95762f39"). InnerVolumeSpecName "kube-api-access-xvgqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:30:32.872833 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:32.872791    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:30:32.925976 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:32.925943    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:30:32.930259 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:32.930241    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvgqq\" (UniqueName: \"kubernetes.io/projected/5b5d4fd5-d238-4dbd-8526-6a0c95762f39-kube-api-access-xvgqq\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:30:32.930341 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:32.930260    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5d4fd5-d238-4dbd-8526-6a0c95762f39-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:30:32.989701 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:32.989681    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af900ad3-aed8-4f48-b718-b70ee90c1972" path="/var/lib/kubelet/pods/af900ad3-aed8-4f48-b718-b70ee90c1972/volumes"
Sep 09 04:30:33.110879 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:33.110818    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:30:33.440382 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:33.440325    2553 generic.go:356] "Generic (PLEG): container finished" podID="5b5d4fd5-d238-4dbd-8526-6a0c95762f39" containerID="9e6c7174106c5c377cc28283c05ba278d3acd83cfe7d2d86f23de04135d136ca" exitCode=0
Sep 09 04:30:33.440382 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:33.440366    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"
Sep 09 04:30:33.440559 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:33.440410    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz" event={"ID":"5b5d4fd5-d238-4dbd-8526-6a0c95762f39","Type":"ContainerDied","Data":"9e6c7174106c5c377cc28283c05ba278d3acd83cfe7d2d86f23de04135d136ca"}
Sep 09 04:30:33.440559 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:33.440472    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz" event={"ID":"5b5d4fd5-d238-4dbd-8526-6a0c95762f39","Type":"ContainerDied","Data":"098d998fc90a2fdac804248515c5ed1e42f867783bc453a6aa78a80286003fb8"}
Sep 09 04:30:33.440559 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:33.440494    2553 scope.go:117] "RemoveContainer" containerID="9e6c7174106c5c377cc28283c05ba278d3acd83cfe7d2d86f23de04135d136ca"
Sep 09 04:30:33.449370 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:33.449351    2553 scope.go:117] "RemoveContainer" containerID="9e6c7174106c5c377cc28283c05ba278d3acd83cfe7d2d86f23de04135d136ca"
Sep 09 04:30:33.449977 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:30:33.449956    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6c7174106c5c377cc28283c05ba278d3acd83cfe7d2d86f23de04135d136ca\": container with ID starting with 9e6c7174106c5c377cc28283c05ba278d3acd83cfe7d2d86f23de04135d136ca not found: ID does not exist" containerID="9e6c7174106c5c377cc28283c05ba278d3acd83cfe7d2d86f23de04135d136ca"
Sep 09 04:30:33.450073 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:33.449982    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6c7174106c5c377cc28283c05ba278d3acd83cfe7d2d86f23de04135d136ca"} err="failed to get container status \"9e6c7174106c5c377cc28283c05ba278d3acd83cfe7d2d86f23de04135d136ca\": rpc error: code = NotFound desc = could not find container \"9e6c7174106c5c377cc28283c05ba278d3acd83cfe7d2d86f23de04135d136ca\": container with ID starting with 9e6c7174106c5c377cc28283c05ba278d3acd83cfe7d2d86f23de04135d136ca not found: ID does not exist"
Sep 09 04:30:33.459424 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:33.459401    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"]
Sep 09 04:30:33.463915 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:33.463895    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ce90-7dcc95d687-q7zzz"]
Sep 09 04:30:34.989400 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:34.989367    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b5d4fd5-d238-4dbd-8526-6a0c95762f39" path="/var/lib/kubelet/pods/5b5d4fd5-d238-4dbd-8526-6a0c95762f39/volumes"
Sep 09 04:30:38.094799 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:38.094769    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"]
Sep 09 04:30:38.140581 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:38.140517    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:30:44.077681 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:44.077645    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:30:53.115672 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.115628    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc" podUID="d3401d03-9be4-4483-bd6d-cdda7d171629" containerName="main" containerID="cri-o://705f8529cfe149bd90cc6bf6e2377e3b777f61de1fcfc8ace82aa934f5759e25" gracePeriod=45
Sep 09 04:30:53.363392 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.363364    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:30:53.510677 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.510653    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8v6r\" (UniqueName: \"kubernetes.io/projected/d3401d03-9be4-4483-bd6d-cdda7d171629-kube-api-access-s8v6r\") pod \"d3401d03-9be4-4483-bd6d-cdda7d171629\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") "
Sep 09 04:30:53.510844 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.510696    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-tmp-dir\") pod \"d3401d03-9be4-4483-bd6d-cdda7d171629\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") "
Sep 09 04:30:53.510844 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.510721    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-dshm\") pod \"d3401d03-9be4-4483-bd6d-cdda7d171629\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") "
Sep 09 04:30:53.510844 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.510751    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-home\") pod \"d3401d03-9be4-4483-bd6d-cdda7d171629\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") "
Sep 09 04:30:53.510844 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.510770    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3401d03-9be4-4483-bd6d-cdda7d171629-tls-certs\") pod \"d3401d03-9be4-4483-bd6d-cdda7d171629\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") "
Sep 09 04:30:53.510844 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.510818    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-model-cache\") pod \"d3401d03-9be4-4483-bd6d-cdda7d171629\" (UID: \"d3401d03-9be4-4483-bd6d-cdda7d171629\") "
Sep 09 04:30:53.511117 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.510921    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3401d03-9be4-4483-bd6d-cdda7d171629" (UID: "d3401d03-9be4-4483-bd6d-cdda7d171629"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:30:53.511117 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.511054    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-home" (OuterVolumeSpecName: "home") pod "d3401d03-9be4-4483-bd6d-cdda7d171629" (UID: "d3401d03-9be4-4483-bd6d-cdda7d171629"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:30:53.511212 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.511115    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3401d03-9be4-4483-bd6d-cdda7d171629" (UID: "d3401d03-9be4-4483-bd6d-cdda7d171629"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:30:53.511251 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.511222    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:30:53.511251 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.511241    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:30:53.511325 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.511254    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:30:53.513056 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.513021    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-dshm" (OuterVolumeSpecName: "dshm") pod "d3401d03-9be4-4483-bd6d-cdda7d171629" (UID: "d3401d03-9be4-4483-bd6d-cdda7d171629"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:30:53.513170 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.513088    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3401d03-9be4-4483-bd6d-cdda7d171629-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3401d03-9be4-4483-bd6d-cdda7d171629" (UID: "d3401d03-9be4-4483-bd6d-cdda7d171629"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:30:53.513170 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.513114    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3401d03-9be4-4483-bd6d-cdda7d171629-kube-api-access-s8v6r" (OuterVolumeSpecName: "kube-api-access-s8v6r") pod "d3401d03-9be4-4483-bd6d-cdda7d171629" (UID: "d3401d03-9be4-4483-bd6d-cdda7d171629"). InnerVolumeSpecName "kube-api-access-s8v6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:30:53.553403 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.553371    2553 generic.go:356] "Generic (PLEG): container finished" podID="d3401d03-9be4-4483-bd6d-cdda7d171629" containerID="705f8529cfe149bd90cc6bf6e2377e3b777f61de1fcfc8ace82aa934f5759e25" exitCode=0
Sep 09 04:30:53.553539 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.553419    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"
Sep 09 04:30:53.553539 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.553474    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc" event={"ID":"d3401d03-9be4-4483-bd6d-cdda7d171629","Type":"ContainerDied","Data":"705f8529cfe149bd90cc6bf6e2377e3b777f61de1fcfc8ace82aa934f5759e25"}
Sep 09 04:30:53.553539 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.553516    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc" event={"ID":"d3401d03-9be4-4483-bd6d-cdda7d171629","Type":"ContainerDied","Data":"88544e1f75e749ca441232f5ff0948055665288d4096b1a92e851ddbcdb53fc0"}
Sep 09 04:30:53.553539 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.553534    2553 scope.go:117] "RemoveContainer" containerID="705f8529cfe149bd90cc6bf6e2377e3b777f61de1fcfc8ace82aa934f5759e25"
Sep 09 04:30:53.565272 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.565249    2553 scope.go:117] "RemoveContainer" containerID="705f8529cfe149bd90cc6bf6e2377e3b777f61de1fcfc8ace82aa934f5759e25"
Sep 09 04:30:53.565571 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:30:53.565554    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705f8529cfe149bd90cc6bf6e2377e3b777f61de1fcfc8ace82aa934f5759e25\": container with ID starting with 705f8529cfe149bd90cc6bf6e2377e3b777f61de1fcfc8ace82aa934f5759e25 not found: ID does not exist" containerID="705f8529cfe149bd90cc6bf6e2377e3b777f61de1fcfc8ace82aa934f5759e25"
Sep 09 04:30:53.565629 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.565579    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705f8529cfe149bd90cc6bf6e2377e3b777f61de1fcfc8ace82aa934f5759e25"} err="failed to get container status \"705f8529cfe149bd90cc6bf6e2377e3b777f61de1fcfc8ace82aa934f5759e25\": rpc error: code = NotFound desc = could not find container \"705f8529cfe149bd90cc6bf6e2377e3b777f61de1fcfc8ace82aa934f5759e25\": container with ID starting with 705f8529cfe149bd90cc6bf6e2377e3b777f61de1fcfc8ace82aa934f5759e25 not found: ID does not exist"
Sep 09 04:30:53.575460 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.575418    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"]
Sep 09 04:30:53.579541 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.579519    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a3ce90/ipp-route-a3ce90-kserve-59fdcd9995-v6hcc"]
Sep 09 04:30:53.612349 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.612331    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8v6r\" (UniqueName: \"kubernetes.io/projected/d3401d03-9be4-4483-bd6d-cdda7d171629-kube-api-access-s8v6r\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:30:53.612431 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.612352    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3401d03-9be4-4483-bd6d-cdda7d171629-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:30:53.612431 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:53.612366    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3401d03-9be4-4483-bd6d-cdda7d171629-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:30:54.988574 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:30:54.988537    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3401d03-9be4-4483-bd6d-cdda7d171629" path="/var/lib/kubelet/pods/d3401d03-9be4-4483-bd6d-cdda7d171629/volumes"
Sep 09 04:31:06.943892 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:06.943860    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:14.238039 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:14.237956    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:15.643027 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:15.642992    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:21.574452 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:21.574403    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:21.640754 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:21.640724    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:21.845713 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:21.845679    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:22.219082 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.219006    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf"]
Sep 09 04:31:22.219536 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.219488    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3401d03-9be4-4483-bd6d-cdda7d171629" containerName="main"
Sep 09 04:31:22.219536 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.219508    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3401d03-9be4-4483-bd6d-cdda7d171629" containerName="main"
Sep 09 04:31:22.219536 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.219528    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5d4fd5-d238-4dbd-8526-6a0c95762f39" containerName="payload-processing"
Sep 09 04:31:22.219536 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.219536    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5d4fd5-d238-4dbd-8526-6a0c95762f39" containerName="payload-processing"
Sep 09 04:31:22.219699 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.219555    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af900ad3-aed8-4f48-b718-b70ee90c1972" containerName="revoke-keys"
Sep 09 04:31:22.219699 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.219563    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="af900ad3-aed8-4f48-b718-b70ee90c1972" containerName="revoke-keys"
Sep 09 04:31:22.219699 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.219693    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5d4fd5-d238-4dbd-8526-6a0c95762f39" containerName="payload-processing"
Sep 09 04:31:22.219793 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.219713    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3401d03-9be4-4483-bd6d-cdda7d171629" containerName="main"
Sep 09 04:31:22.219793 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.219722    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="af900ad3-aed8-4f48-b718-b70ee90c1972" containerName="revoke-keys"
Sep 09 04:31:22.224253 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.224234    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf"
Sep 09 04:31:22.229187 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.229161    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:31:22.229350 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.229333    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:31:22.233793 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.233769    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf"]
Sep 09 04:31:22.249572 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.249544    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8670c9-6170-4865-baac-a5210e4d8792-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf\" (UID: \"fc8670c9-6170-4865-baac-a5210e4d8792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf"
Sep 09 04:31:22.350279 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.350251    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8670c9-6170-4865-baac-a5210e4d8792-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf\" (UID: \"fc8670c9-6170-4865-baac-a5210e4d8792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf"
Sep 09 04:31:22.350953 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.350933    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8670c9-6170-4865-baac-a5210e4d8792-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf\" (UID: \"fc8670c9-6170-4865-baac-a5210e4d8792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf"
Sep 09 04:31:22.388633 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.388605    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:22.535234 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.535207    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf"
Sep 09 04:31:22.808849 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:22.808825    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf"]
Sep 09 04:31:22.811133 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:31:22.811108    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8670c9_6170_4865_baac_a5210e4d8792.slice/crio-06e6809185dd16152d9c60c4b938354f2d97f43f235cdace3462dd5439e439da WatchSource:0}: Error finding container 06e6809185dd16152d9c60c4b938354f2d97f43f235cdace3462dd5439e439da: Status 404 returned error can't find the container with id 06e6809185dd16152d9c60c4b938354f2d97f43f235cdace3462dd5439e439da
Sep 09 04:31:23.716479 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:23.716403    2553 generic.go:356] "Generic (PLEG): container finished" podID="fc8670c9-6170-4865-baac-a5210e4d8792" containerID="d4bb1aed5896e61051735327104b433e9685f9fe936ea6ec5da12b42a70c4eb2" exitCode=0
Sep 09 04:31:23.716479 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:23.716473    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf" event={"ID":"fc8670c9-6170-4865-baac-a5210e4d8792","Type":"ContainerDied","Data":"d4bb1aed5896e61051735327104b433e9685f9fe936ea6ec5da12b42a70c4eb2"}
Sep 09 04:31:23.716620 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:23.716518    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf" event={"ID":"fc8670c9-6170-4865-baac-a5210e4d8792","Type":"ContainerStarted","Data":"06e6809185dd16152d9c60c4b938354f2d97f43f235cdace3462dd5439e439da"}
Sep 09 04:31:24.853431 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:24.853407    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf"
Sep 09 04:31:24.973186 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:24.973117    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8670c9-6170-4865-baac-a5210e4d8792-maas-api-service-ca\") pod \"fc8670c9-6170-4865-baac-a5210e4d8792\" (UID: \"fc8670c9-6170-4865-baac-a5210e4d8792\") "
Sep 09 04:31:24.973523 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:24.973500    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8670c9-6170-4865-baac-a5210e4d8792-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fc8670c9-6170-4865-baac-a5210e4d8792" (UID: "fc8670c9-6170-4865-baac-a5210e4d8792"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:31:25.074487 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:25.074464    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8670c9-6170-4865-baac-a5210e4d8792-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:31:25.728986 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:25.728949    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf" event={"ID":"fc8670c9-6170-4865-baac-a5210e4d8792","Type":"ContainerDied","Data":"06e6809185dd16152d9c60c4b938354f2d97f43f235cdace3462dd5439e439da"}
Sep 09 04:31:25.729151 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:25.729001    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06e6809185dd16152d9c60c4b938354f2d97f43f235cdace3462dd5439e439da"
Sep 09 04:31:25.729151 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:25.728966    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf"
Sep 09 04:31:32.744290 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:32.744252    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0db87d-29815470-m6dzs"]
Sep 09 04:31:32.746422 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:32.746399    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0db87d-29815470-m6dzs"]
Sep 09 04:31:32.764080 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:32.764051    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:32.776598 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:32.776569    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"]
Sep 09 04:31:32.776805 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:32.776778    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8" podUID="9eb2561a-0426-4a68-aa41-72ef8985b4e2" containerName="payload-pre-processing" containerID="cri-o://9874975cef52843681dc74c4d76da8e4d4eb3e50c0778a07c2fc274d23faac53" gracePeriod=30
Sep 09 04:31:32.860867 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:32.860837    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:32.940878 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:32.940838    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf"]
Sep 09 04:31:32.943229 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:32.943203    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db87d-h5stf"]
Sep 09 04:31:32.943467 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:32.943415    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:32.989839 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:32.989755    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e44b1cb2-c8e5-4ce5-9b7a-87c7b23b44f9" path="/var/lib/kubelet/pods/e44b1cb2-c8e5-4ce5-9b7a-87c7b23b44f9/volumes"
Sep 09 04:31:32.990172 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:32.990149    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc8670c9-6170-4865-baac-a5210e4d8792" path="/var/lib/kubelet/pods/fc8670c9-6170-4865-baac-a5210e4d8792/volumes"
Sep 09 04:31:32.999683 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:32.999625    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:33.035490 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.035469    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"
Sep 09 04:31:33.144787 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.144753    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pclp\" (UniqueName: \"kubernetes.io/projected/9eb2561a-0426-4a68-aa41-72ef8985b4e2-kube-api-access-5pclp\") pod \"9eb2561a-0426-4a68-aa41-72ef8985b4e2\" (UID: \"9eb2561a-0426-4a68-aa41-72ef8985b4e2\") "
Sep 09 04:31:33.144787 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.144792    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb2561a-0426-4a68-aa41-72ef8985b4e2-plugins-config-volume\") pod \"9eb2561a-0426-4a68-aa41-72ef8985b4e2\" (UID: \"9eb2561a-0426-4a68-aa41-72ef8985b4e2\") "
Sep 09 04:31:33.145203 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.145179    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb2561a-0426-4a68-aa41-72ef8985b4e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9eb2561a-0426-4a68-aa41-72ef8985b4e2" (UID: "9eb2561a-0426-4a68-aa41-72ef8985b4e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:31:33.147132 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.147109    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb2561a-0426-4a68-aa41-72ef8985b4e2-kube-api-access-5pclp" (OuterVolumeSpecName: "kube-api-access-5pclp") pod "9eb2561a-0426-4a68-aa41-72ef8985b4e2" (UID: "9eb2561a-0426-4a68-aa41-72ef8985b4e2"). InnerVolumeSpecName "kube-api-access-5pclp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:31:33.246020 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.245996    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pclp\" (UniqueName: \"kubernetes.io/projected/9eb2561a-0426-4a68-aa41-72ef8985b4e2-kube-api-access-5pclp\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:31:33.246020 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.246018    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb2561a-0426-4a68-aa41-72ef8985b4e2-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:31:33.331146 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.331061    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:33.775564 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.775535    2553 generic.go:356] "Generic (PLEG): container finished" podID="9eb2561a-0426-4a68-aa41-72ef8985b4e2" containerID="9874975cef52843681dc74c4d76da8e4d4eb3e50c0778a07c2fc274d23faac53" exitCode=0
Sep 09 04:31:33.775934 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.775586    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"
Sep 09 04:31:33.775934 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.775612    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8" event={"ID":"9eb2561a-0426-4a68-aa41-72ef8985b4e2","Type":"ContainerDied","Data":"9874975cef52843681dc74c4d76da8e4d4eb3e50c0778a07c2fc274d23faac53"}
Sep 09 04:31:33.775934 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.775681    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8" event={"ID":"9eb2561a-0426-4a68-aa41-72ef8985b4e2","Type":"ContainerDied","Data":"84a9dbdc338e81193d1881bdc23a1b61b55a96d99961bc4c2b435739b5313f66"}
Sep 09 04:31:33.775934 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.775704    2553 scope.go:117] "RemoveContainer" containerID="9874975cef52843681dc74c4d76da8e4d4eb3e50c0778a07c2fc274d23faac53"
Sep 09 04:31:33.785346 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.785148    2553 scope.go:117] "RemoveContainer" containerID="9874975cef52843681dc74c4d76da8e4d4eb3e50c0778a07c2fc274d23faac53"
Sep 09 04:31:33.785504 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:31:33.785463    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9874975cef52843681dc74c4d76da8e4d4eb3e50c0778a07c2fc274d23faac53\": container with ID starting with 9874975cef52843681dc74c4d76da8e4d4eb3e50c0778a07c2fc274d23faac53 not found: ID does not exist" containerID="9874975cef52843681dc74c4d76da8e4d4eb3e50c0778a07c2fc274d23faac53"
Sep 09 04:31:33.785504 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.785489    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9874975cef52843681dc74c4d76da8e4d4eb3e50c0778a07c2fc274d23faac53"} err="failed to get container status \"9874975cef52843681dc74c4d76da8e4d4eb3e50c0778a07c2fc274d23faac53\": rpc error: code = NotFound desc = could not find container \"9874975cef52843681dc74c4d76da8e4d4eb3e50c0778a07c2fc274d23faac53\": container with ID starting with 9874975cef52843681dc74c4d76da8e4d4eb3e50c0778a07c2fc274d23faac53 not found: ID does not exist"
Sep 09 04:31:33.797370 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.797349    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"]
Sep 09 04:31:33.804026 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:33.804007    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db87d-7688767f8b-jsvh8"]
Sep 09 04:31:34.989006 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:34.988975    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eb2561a-0426-4a68-aa41-72ef8985b4e2" path="/var/lib/kubelet/pods/9eb2561a-0426-4a68-aa41-72ef8985b4e2/volumes"
Sep 09 04:31:36.671876 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:36.671843    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:31:39.579707 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:39.579674    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:39.684133 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:39.684103    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:39.871211 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:39.871127    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:40.211385 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.211314    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb"]
Sep 09 04:31:40.211738 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.211725    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8670c9-6170-4865-baac-a5210e4d8792" containerName="revoke-keys"
Sep 09 04:31:40.211783 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.211740    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8670c9-6170-4865-baac-a5210e4d8792" containerName="revoke-keys"
Sep 09 04:31:40.211783 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.211749    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb2561a-0426-4a68-aa41-72ef8985b4e2" containerName="payload-pre-processing"
Sep 09 04:31:40.211783 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.211754    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb2561a-0426-4a68-aa41-72ef8985b4e2" containerName="payload-pre-processing"
Sep 09 04:31:40.211878 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.211835    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb2561a-0426-4a68-aa41-72ef8985b4e2" containerName="payload-pre-processing"
Sep 09 04:31:40.211878 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.211844    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8670c9-6170-4865-baac-a5210e4d8792" containerName="revoke-keys"
Sep 09 04:31:40.219107 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.218464    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb"
Sep 09 04:31:40.221542 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.221518    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:31:40.222209 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.222188    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:31:40.224245 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.224221    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb"]
Sep 09 04:31:40.310455 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.310422    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e05a76c7-53db-4c02-9097-b800da5a7676-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb\" (UID: \"e05a76c7-53db-4c02-9097-b800da5a7676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb"
Sep 09 04:31:40.411885 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.411862    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e05a76c7-53db-4c02-9097-b800da5a7676-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb\" (UID: \"e05a76c7-53db-4c02-9097-b800da5a7676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb"
Sep 09 04:31:40.412421 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.412404    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e05a76c7-53db-4c02-9097-b800da5a7676-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb\" (UID: \"e05a76c7-53db-4c02-9097-b800da5a7676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb"
Sep 09 04:31:40.425534 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.425487    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:40.531187 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.531162    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb"
Sep 09 04:31:40.804298 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.804272    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb"]
Sep 09 04:31:40.806389 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:31:40.806363    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05a76c7_53db_4c02_9097_b800da5a7676.slice/crio-37363e75f89f013a990775026c8ec751bc2189ce00c46b01df9c6256967f3a3c WatchSource:0}: Error finding container 37363e75f89f013a990775026c8ec751bc2189ce00c46b01df9c6256967f3a3c: Status 404 returned error can't find the container with id 37363e75f89f013a990775026c8ec751bc2189ce00c46b01df9c6256967f3a3c
Sep 09 04:31:40.815553 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:40.815529    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb" event={"ID":"e05a76c7-53db-4c02-9097-b800da5a7676","Type":"ContainerStarted","Data":"37363e75f89f013a990775026c8ec751bc2189ce00c46b01df9c6256967f3a3c"}
Sep 09 04:31:41.822124 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:41.822096    2553 generic.go:356] "Generic (PLEG): container finished" podID="e05a76c7-53db-4c02-9097-b800da5a7676" containerID="912c620145e5823c2b2ecb0e6e62b1074526e09f734ec19b2bff31df88d6d501" exitCode=0
Sep 09 04:31:41.822488 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:41.822135    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb" event={"ID":"e05a76c7-53db-4c02-9097-b800da5a7676","Type":"ContainerDied","Data":"912c620145e5823c2b2ecb0e6e62b1074526e09f734ec19b2bff31df88d6d501"}
Sep 09 04:31:42.968887 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:42.968859    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb"
Sep 09 04:31:43.134570 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:43.134501    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e05a76c7-53db-4c02-9097-b800da5a7676-maas-api-service-ca\") pod \"e05a76c7-53db-4c02-9097-b800da5a7676\" (UID: \"e05a76c7-53db-4c02-9097-b800da5a7676\") "
Sep 09 04:31:43.134815 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:43.134792    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e05a76c7-53db-4c02-9097-b800da5a7676-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e05a76c7-53db-4c02-9097-b800da5a7676" (UID: "e05a76c7-53db-4c02-9097-b800da5a7676"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:31:43.235959 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:43.235930    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e05a76c7-53db-4c02-9097-b800da5a7676-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:31:43.841492 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:43.841437    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb"
Sep 09 04:31:43.841492 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:43.841468    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb" event={"ID":"e05a76c7-53db-4c02-9097-b800da5a7676","Type":"ContainerDied","Data":"37363e75f89f013a990775026c8ec751bc2189ce00c46b01df9c6256967f3a3c"}
Sep 09 04:31:43.841705 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:43.841553    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37363e75f89f013a990775026c8ec751bc2189ce00c46b01df9c6256967f3a3c"
Sep 09 04:31:50.857737 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:50.857698    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:50.894282 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:50.894249    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"]
Sep 09 04:31:50.894582 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:50.894527    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz" podUID="3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c" containerName="payload-pre-processing" containerID="cri-o://1817eb34a4647c814d525d574df6b7dc107643ce1e9cf604b1e565a2050a0caf" gracePeriod=30
Sep 09 04:31:50.917222 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:50.917142    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:50.982610 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:50.982577    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:51.069483 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.069424    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb"]
Sep 09 04:31:51.073306 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.073241    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:51.075100 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.075077    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-426183-rfgcb"]
Sep 09 04:31:51.131808 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.131778    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:51.149724 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.149703    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"
Sep 09 04:31:51.302194 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.302163    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c-plugins-config-volume\") pod \"3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c\" (UID: \"3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c\") "
Sep 09 04:31:51.302350 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.302201    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlx4l\" (UniqueName: \"kubernetes.io/projected/3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c-kube-api-access-hlx4l\") pod \"3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c\" (UID: \"3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c\") "
Sep 09 04:31:51.302559 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.302536    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c" (UID: "3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:31:51.304372 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.304346    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c-kube-api-access-hlx4l" (OuterVolumeSpecName: "kube-api-access-hlx4l") pod "3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c" (UID: "3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c"). InnerVolumeSpecName "kube-api-access-hlx4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:31:51.403192 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.403126    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlx4l\" (UniqueName: \"kubernetes.io/projected/3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c-kube-api-access-hlx4l\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:31:51.403192 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.403152    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:31:51.433548 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.433516    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:51.885884 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.885854    2553 generic.go:356] "Generic (PLEG): container finished" podID="3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c" containerID="1817eb34a4647c814d525d574df6b7dc107643ce1e9cf604b1e565a2050a0caf" exitCode=0
Sep 09 04:31:51.886237 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.885893    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"
Sep 09 04:31:51.886237 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.885942    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz" event={"ID":"3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c","Type":"ContainerDied","Data":"1817eb34a4647c814d525d574df6b7dc107643ce1e9cf604b1e565a2050a0caf"}
Sep 09 04:31:51.886237 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.886022    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz" event={"ID":"3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c","Type":"ContainerDied","Data":"a1acd03affebf44d43a8e14390ea4ad10ed1f72157b5e83d1930f5c52a4cda29"}
Sep 09 04:31:51.886237 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.886047    2553 scope.go:117] "RemoveContainer" containerID="1817eb34a4647c814d525d574df6b7dc107643ce1e9cf604b1e565a2050a0caf"
Sep 09 04:31:51.895604 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.895414    2553 scope.go:117] "RemoveContainer" containerID="1817eb34a4647c814d525d574df6b7dc107643ce1e9cf604b1e565a2050a0caf"
Sep 09 04:31:51.895730 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:31:51.895712    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1817eb34a4647c814d525d574df6b7dc107643ce1e9cf604b1e565a2050a0caf\": container with ID starting with 1817eb34a4647c814d525d574df6b7dc107643ce1e9cf604b1e565a2050a0caf not found: ID does not exist" containerID="1817eb34a4647c814d525d574df6b7dc107643ce1e9cf604b1e565a2050a0caf"
Sep 09 04:31:51.895777 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.895739    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1817eb34a4647c814d525d574df6b7dc107643ce1e9cf604b1e565a2050a0caf"} err="failed to get container status \"1817eb34a4647c814d525d574df6b7dc107643ce1e9cf604b1e565a2050a0caf\": rpc error: code = NotFound desc = could not find container \"1817eb34a4647c814d525d574df6b7dc107643ce1e9cf604b1e565a2050a0caf\": container with ID starting with 1817eb34a4647c814d525d574df6b7dc107643ce1e9cf604b1e565a2050a0caf not found: ID does not exist"
Sep 09 04:31:51.908330 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.908310    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"]
Sep 09 04:31:51.912561 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:51.912542    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-426183-f8c6b7fcd-rwvsz"]
Sep 09 04:31:52.989736 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:52.989706    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c" path="/var/lib/kubelet/pods/3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c/volumes"
Sep 09 04:31:52.990105 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:52.990006    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e05a76c7-53db-4c02-9097-b800da5a7676" path="/var/lib/kubelet/pods/e05a76c7-53db-4c02-9097-b800da5a7676/volumes"
Sep 09 04:31:54.169530 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:54.169498    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:31:57.866997 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:57.866966    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:57.967536 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:57.967499    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:58.139531 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.139435    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:58.490041 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.490010    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9"]
Sep 09 04:31:58.490470 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.490456    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e05a76c7-53db-4c02-9097-b800da5a7676" containerName="revoke-keys"
Sep 09 04:31:58.490538 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.490474    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e05a76c7-53db-4c02-9097-b800da5a7676" containerName="revoke-keys"
Sep 09 04:31:58.490538 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.490499    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c" containerName="payload-pre-processing"
Sep 09 04:31:58.490538 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.490504    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c" containerName="payload-pre-processing"
Sep 09 04:31:58.490636 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.490571    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e05a76c7-53db-4c02-9097-b800da5a7676" containerName="revoke-keys"
Sep 09 04:31:58.490636 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.490582    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b1fe41b-0c5a-4b6e-ac64-fc9bdf45952c" containerName="payload-pre-processing"
Sep 09 04:31:58.493650 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.493635    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9"
Sep 09 04:31:58.496109 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.496047    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:31:58.496109 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.496088    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:31:58.501042 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.501006    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9"]
Sep 09 04:31:58.566242 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.566216    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c59425-16df-47e8-9fd0-944d21489318-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9\" (UID: \"b1c59425-16df-47e8-9fd0-944d21489318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9"
Sep 09 04:31:58.602467 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.602420    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:31:58.667163 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.667137    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c59425-16df-47e8-9fd0-944d21489318-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9\" (UID: \"b1c59425-16df-47e8-9fd0-944d21489318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9"
Sep 09 04:31:58.667794 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.667773    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c59425-16df-47e8-9fd0-944d21489318-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9\" (UID: \"b1c59425-16df-47e8-9fd0-944d21489318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9"
Sep 09 04:31:58.806676 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:58.806610    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9"
Sep 09 04:31:59.081520 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:59.081498    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9"]
Sep 09 04:31:59.083507 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:31:59.083478    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c59425_16df_47e8_9fd0_944d21489318.slice/crio-e897c05e948b4134260c59ffaed7891830466477880240643cbe29e223f73f0d WatchSource:0}: Error finding container e897c05e948b4134260c59ffaed7891830466477880240643cbe29e223f73f0d: Status 404 returned error can't find the container with id e897c05e948b4134260c59ffaed7891830466477880240643cbe29e223f73f0d
Sep 09 04:31:59.930612 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:59.930580    2553 generic.go:356] "Generic (PLEG): container finished" podID="b1c59425-16df-47e8-9fd0-944d21489318" containerID="fb037bde3ec8ca7e1c8d81d91417bc9b286b3c69963eb895308f8732ee956e8d" exitCode=0
Sep 09 04:31:59.930771 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:59.930624    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9" event={"ID":"b1c59425-16df-47e8-9fd0-944d21489318","Type":"ContainerDied","Data":"fb037bde3ec8ca7e1c8d81d91417bc9b286b3c69963eb895308f8732ee956e8d"}
Sep 09 04:31:59.930771 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:31:59.930663    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9" event={"ID":"b1c59425-16df-47e8-9fd0-944d21489318","Type":"ContainerStarted","Data":"e897c05e948b4134260c59ffaed7891830466477880240643cbe29e223f73f0d"}
Sep 09 04:32:01.064341 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:01.064319    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9"
Sep 09 04:32:01.088774 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:01.088752    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c59425-16df-47e8-9fd0-944d21489318-maas-api-service-ca\") pod \"b1c59425-16df-47e8-9fd0-944d21489318\" (UID: \"b1c59425-16df-47e8-9fd0-944d21489318\") "
Sep 09 04:32:01.089081 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:01.089056    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c59425-16df-47e8-9fd0-944d21489318-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1c59425-16df-47e8-9fd0-944d21489318" (UID: "b1c59425-16df-47e8-9fd0-944d21489318"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:32:01.190176 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:01.190111    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c59425-16df-47e8-9fd0-944d21489318-maas-api-service-ca\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:32:01.942311 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:01.942274    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9" event={"ID":"b1c59425-16df-47e8-9fd0-944d21489318","Type":"ContainerDied","Data":"e897c05e948b4134260c59ffaed7891830466477880240643cbe29e223f73f0d"}
Sep 09 04:32:01.942489 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:01.942328    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e897c05e948b4134260c59ffaed7891830466477880240643cbe29e223f73f0d"
Sep 09 04:32:01.942489 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:01.942291    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9"
Sep 09 04:32:08.967009 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:08.966974    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:32:08.970642 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:08.970618    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9c44b5-29815470-xhdzd"]
Sep 09 04:32:08.974405 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:08.974382    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9c44b5-29815470-xhdzd"]
Sep 09 04:32:08.990100 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:08.990071    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e365790-553d-4d9d-9a8f-59eaf55c7524" path="/var/lib/kubelet/pods/2e365790-553d-4d9d-9a8f-59eaf55c7524/volumes"
Sep 09 04:32:08.996600 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:08.996575    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"]
Sep 09 04:32:08.996817 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:08.996793    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs" podUID="d71f8271-7604-4f45-aa3b-0915614d37a9" containerName="payload-processing" containerID="cri-o://37cd8f15e82f7f2a357a8ca2a46e845df058a6f54d61db3f0b5d53d06c5c7c3d" gracePeriod=30
Sep 09 04:32:09.002302 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.002279    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"]
Sep 09 04:32:09.002529 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.002509    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4" podUID="2970d9cb-0b4a-4045-aecb-5e2e34ae8226" containerName="payload-pre-processing" containerID="cri-o://96128450aa026d6f217c0cf18192c192c15429e2ca3e8242ae088b5d4c81e597" gracePeriod=30
Sep 09 04:32:09.032578 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.032548    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:32:09.132281 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.132250    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:32:09.159047 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.158964    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9"]
Sep 09 04:32:09.161581 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.161559    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9c44b5-w5mh9"]
Sep 09 04:32:09.188556 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.188518    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:32:09.279086 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.279054    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"
Sep 09 04:32:09.282182 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.282160    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"
Sep 09 04:32:09.353823 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.353789    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxwlx\" (UniqueName: \"kubernetes.io/projected/d71f8271-7604-4f45-aa3b-0915614d37a9-kube-api-access-hxwlx\") pod \"d71f8271-7604-4f45-aa3b-0915614d37a9\" (UID: \"d71f8271-7604-4f45-aa3b-0915614d37a9\") "
Sep 09 04:32:09.354005 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.353890    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8qlx\" (UniqueName: \"kubernetes.io/projected/2970d9cb-0b4a-4045-aecb-5e2e34ae8226-kube-api-access-c8qlx\") pod \"2970d9cb-0b4a-4045-aecb-5e2e34ae8226\" (UID: \"2970d9cb-0b4a-4045-aecb-5e2e34ae8226\") "
Sep 09 04:32:09.354005 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.353916    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71f8271-7604-4f45-aa3b-0915614d37a9-plugins-config-volume\") pod \"d71f8271-7604-4f45-aa3b-0915614d37a9\" (UID: \"d71f8271-7604-4f45-aa3b-0915614d37a9\") "
Sep 09 04:32:09.354129 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.354013    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2970d9cb-0b4a-4045-aecb-5e2e34ae8226-plugins-config-volume\") pod \"2970d9cb-0b4a-4045-aecb-5e2e34ae8226\" (UID: \"2970d9cb-0b4a-4045-aecb-5e2e34ae8226\") "
Sep 09 04:32:09.354434 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.354407    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71f8271-7604-4f45-aa3b-0915614d37a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d71f8271-7604-4f45-aa3b-0915614d37a9" (UID: "d71f8271-7604-4f45-aa3b-0915614d37a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:32:09.354434 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.354417    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2970d9cb-0b4a-4045-aecb-5e2e34ae8226-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2970d9cb-0b4a-4045-aecb-5e2e34ae8226" (UID: "2970d9cb-0b4a-4045-aecb-5e2e34ae8226"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:32:09.356043 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.356020    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71f8271-7604-4f45-aa3b-0915614d37a9-kube-api-access-hxwlx" (OuterVolumeSpecName: "kube-api-access-hxwlx") pod "d71f8271-7604-4f45-aa3b-0915614d37a9" (UID: "d71f8271-7604-4f45-aa3b-0915614d37a9"). InnerVolumeSpecName "kube-api-access-hxwlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:32:09.356109 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.356035    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2970d9cb-0b4a-4045-aecb-5e2e34ae8226-kube-api-access-c8qlx" (OuterVolumeSpecName: "kube-api-access-c8qlx") pod "2970d9cb-0b4a-4045-aecb-5e2e34ae8226" (UID: "2970d9cb-0b4a-4045-aecb-5e2e34ae8226"). InnerVolumeSpecName "kube-api-access-c8qlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:32:09.454943 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.454919    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2970d9cb-0b4a-4045-aecb-5e2e34ae8226-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:32:09.454943 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.454942    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxwlx\" (UniqueName: \"kubernetes.io/projected/d71f8271-7604-4f45-aa3b-0915614d37a9-kube-api-access-hxwlx\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:32:09.455079 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.454957    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8qlx\" (UniqueName: \"kubernetes.io/projected/2970d9cb-0b4a-4045-aecb-5e2e34ae8226-kube-api-access-c8qlx\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:32:09.455079 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.454969    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71f8271-7604-4f45-aa3b-0915614d37a9-plugins-config-volume\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:32:09.523932 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.523876    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:32:09.986804 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.986772    2553 generic.go:356] "Generic (PLEG): container finished" podID="2970d9cb-0b4a-4045-aecb-5e2e34ae8226" containerID="96128450aa026d6f217c0cf18192c192c15429e2ca3e8242ae088b5d4c81e597" exitCode=0
Sep 09 04:32:09.987216 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.986816    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"
Sep 09 04:32:09.987216 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.986854    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4" event={"ID":"2970d9cb-0b4a-4045-aecb-5e2e34ae8226","Type":"ContainerDied","Data":"96128450aa026d6f217c0cf18192c192c15429e2ca3e8242ae088b5d4c81e597"}
Sep 09 04:32:09.987216 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.986895    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4" event={"ID":"2970d9cb-0b4a-4045-aecb-5e2e34ae8226","Type":"ContainerDied","Data":"90aafe14ae83716a16f2fb1a28c493205d71d8fecbfb02ced76ebcf4d1cb7645"}
Sep 09 04:32:09.987216 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.986916    2553 scope.go:117] "RemoveContainer" containerID="96128450aa026d6f217c0cf18192c192c15429e2ca3e8242ae088b5d4c81e597"
Sep 09 04:32:09.988340 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.988320    2553 generic.go:356] "Generic (PLEG): container finished" podID="d71f8271-7604-4f45-aa3b-0915614d37a9" containerID="37cd8f15e82f7f2a357a8ca2a46e845df058a6f54d61db3f0b5d53d06c5c7c3d" exitCode=0
Sep 09 04:32:09.988431 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.988365    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"
Sep 09 04:32:09.988431 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.988386    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs" event={"ID":"d71f8271-7604-4f45-aa3b-0915614d37a9","Type":"ContainerDied","Data":"37cd8f15e82f7f2a357a8ca2a46e845df058a6f54d61db3f0b5d53d06c5c7c3d"}
Sep 09 04:32:09.988431 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.988425    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs" event={"ID":"d71f8271-7604-4f45-aa3b-0915614d37a9","Type":"ContainerDied","Data":"414ec6af565035ef10f732873160b54625947a61b33049ed018f0af193cbf9f4"}
Sep 09 04:32:09.998053 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.997865    2553 scope.go:117] "RemoveContainer" containerID="96128450aa026d6f217c0cf18192c192c15429e2ca3e8242ae088b5d4c81e597"
Sep 09 04:32:09.998292 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:32:09.998185    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96128450aa026d6f217c0cf18192c192c15429e2ca3e8242ae088b5d4c81e597\": container with ID starting with 96128450aa026d6f217c0cf18192c192c15429e2ca3e8242ae088b5d4c81e597 not found: ID does not exist" containerID="96128450aa026d6f217c0cf18192c192c15429e2ca3e8242ae088b5d4c81e597"
Sep 09 04:32:09.998292 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.998219    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96128450aa026d6f217c0cf18192c192c15429e2ca3e8242ae088b5d4c81e597"} err="failed to get container status \"96128450aa026d6f217c0cf18192c192c15429e2ca3e8242ae088b5d4c81e597\": rpc error: code = NotFound desc = could not find container \"96128450aa026d6f217c0cf18192c192c15429e2ca3e8242ae088b5d4c81e597\": container with ID starting with 96128450aa026d6f217c0cf18192c192c15429e2ca3e8242ae088b5d4c81e597 not found: ID does not exist"
Sep 09 04:32:09.998292 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:09.998246    2553 scope.go:117] "RemoveContainer" containerID="37cd8f15e82f7f2a357a8ca2a46e845df058a6f54d61db3f0b5d53d06c5c7c3d"
Sep 09 04:32:10.006140 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:10.006123    2553 scope.go:117] "RemoveContainer" containerID="37cd8f15e82f7f2a357a8ca2a46e845df058a6f54d61db3f0b5d53d06c5c7c3d"
Sep 09 04:32:10.006411 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:32:10.006369    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37cd8f15e82f7f2a357a8ca2a46e845df058a6f54d61db3f0b5d53d06c5c7c3d\": container with ID starting with 37cd8f15e82f7f2a357a8ca2a46e845df058a6f54d61db3f0b5d53d06c5c7c3d not found: ID does not exist" containerID="37cd8f15e82f7f2a357a8ca2a46e845df058a6f54d61db3f0b5d53d06c5c7c3d"
Sep 09 04:32:10.006543 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:10.006424    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37cd8f15e82f7f2a357a8ca2a46e845df058a6f54d61db3f0b5d53d06c5c7c3d"} err="failed to get container status \"37cd8f15e82f7f2a357a8ca2a46e845df058a6f54d61db3f0b5d53d06c5c7c3d\": rpc error: code = NotFound desc = could not find container \"37cd8f15e82f7f2a357a8ca2a46e845df058a6f54d61db3f0b5d53d06c5c7c3d\": container with ID starting with 37cd8f15e82f7f2a357a8ca2a46e845df058a6f54d61db3f0b5d53d06c5c7c3d not found: ID does not exist"
Sep 09 04:32:10.009057 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:10.009034    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"]
Sep 09 04:32:10.013408 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:10.013385    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9c44b5-7ddcd84fc8-6v8j4"]
Sep 09 04:32:10.023750 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:10.023731    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"]
Sep 09 04:32:10.027918 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:10.027901    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9c44b5-7c64d99bbc-2fgjs"]
Sep 09 04:32:10.988861 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:10.988820    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2970d9cb-0b4a-4045-aecb-5e2e34ae8226" path="/var/lib/kubelet/pods/2970d9cb-0b4a-4045-aecb-5e2e34ae8226/volumes"
Sep 09 04:32:10.989513 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:10.989284    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1c59425-16df-47e8-9fd0-944d21489318" path="/var/lib/kubelet/pods/b1c59425-16df-47e8-9fd0-944d21489318/volumes"
Sep 09 04:32:10.989757 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:10.989736    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d71f8271-7604-4f45-aa3b-0915614d37a9" path="/var/lib/kubelet/pods/d71f8271-7604-4f45-aa3b-0915614d37a9/volumes"
Sep 09 04:32:11.778162 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:11.778130    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:32:15.999555 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:15.999520    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:32:16.099627 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:16.099599    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:32:16.389571 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:16.389473    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:32:21.155521 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:32:21.155467    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:33:39.258086 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:33:39.258058    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:33:39.767764 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:33:39.767733    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:33:40.471613 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:33:40.471580    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:33:41.781249 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:33:41.781215    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:34:02.000704 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:34:02.000673    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:34:05.662476 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:34:05.662430    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:34:16.360557 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:34:16.360477    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:34:18.241531 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:34:18.241496    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:34:25.158338 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:34:25.158308    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:34:28.816966 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:34:28.816938    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:34:34.756969 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:34:34.756938    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:34:48.663693 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:34:48.663658    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:34:58.463794 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:34:58.463765    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:35:29.161589 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:35:29.161549    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:35:44.318562 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:35:44.318484    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:36:17.371615 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:17.371581    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:36:17.473222 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:17.473174    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:36:21.513266 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:21.513230    2553 scope.go:117] "RemoveContainer" containerID="1b4bb9c14f541aec86d28de774132f4d77e083ffcf0109d771ef9d34047da53d"
Sep 09 04:36:21.521496 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:21.521476    2553 scope.go:117] "RemoveContainer" containerID="38310d4dbbad9004d8d1b99c6d7fcbe0b61149b42a2bec2853121487f3a302a7"
Sep 09 04:36:21.529048 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:21.529032    2553 scope.go:117] "RemoveContainer" containerID="3cfcdc1ae943d8893a27bfb6fbbeb4e840869234d672853bbf1066a8029b7af0"
Sep 09 04:36:21.536618 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:21.536600    2553 scope.go:117] "RemoveContainer" containerID="5abac9166743b7559aabede0050bf51f57748cca2ef686af7bdbb318abf8b6a9"
Sep 09 04:36:21.543950 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:21.543930    2553 scope.go:117] "RemoveContainer" containerID="a93fc2d93373799f61d06c1a9c87b695f87cca5947b08779fc6a46b1064bbd8e"
Sep 09 04:36:28.778453 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:28.778415    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:36:36.776393 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:36.776359    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:36:45.268153 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:45.268118    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:36:46.667414 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.667375    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"]
Sep 09 04:36:46.667828 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.667814    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c59425-16df-47e8-9fd0-944d21489318" containerName="revoke-keys"
Sep 09 04:36:46.667880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.667831    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c59425-16df-47e8-9fd0-944d21489318" containerName="revoke-keys"
Sep 09 04:36:46.667880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.667849    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d71f8271-7604-4f45-aa3b-0915614d37a9" containerName="payload-processing"
Sep 09 04:36:46.667880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.667855    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d71f8271-7604-4f45-aa3b-0915614d37a9" containerName="payload-processing"
Sep 09 04:36:46.667880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.667866    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2970d9cb-0b4a-4045-aecb-5e2e34ae8226" containerName="payload-pre-processing"
Sep 09 04:36:46.667880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.667875    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2970d9cb-0b4a-4045-aecb-5e2e34ae8226" containerName="payload-pre-processing"
Sep 09 04:36:46.668041 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.667964    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d71f8271-7604-4f45-aa3b-0915614d37a9" containerName="payload-processing"
Sep 09 04:36:46.668041 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.667976    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c59425-16df-47e8-9fd0-944d21489318" containerName="revoke-keys"
Sep 09 04:36:46.668041 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.667983    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2970d9cb-0b4a-4045-aecb-5e2e34ae8226" containerName="payload-pre-processing"
Sep 09 04:36:46.671437 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.671422    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.674751 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.674729    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2d97db68-kserve-self-signed-certs\""
Sep 09 04:36:46.681744 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.681716    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"]
Sep 09 04:36:46.705377 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.705356    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:36:46.818923 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.818886    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-home\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.819110 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.819035    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-model-cache\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.819110 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.819065    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-tmp-dir\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.819110 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.819087    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-tls-certs\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.819268 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.819115    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz4fc\" (UniqueName: \"kubernetes.io/projected/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-kube-api-access-tz4fc\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.819268 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.819147    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-dshm\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.920357 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.920293    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-model-cache\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.920357 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.920323    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-tmp-dir\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.920357 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.920338    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-tls-certs\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.920357 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.920355    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4fc\" (UniqueName: \"kubernetes.io/projected/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-kube-api-access-tz4fc\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.920679 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.920375    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-dshm\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.920679 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.920416    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-home\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.920783 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.920760    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-tmp-dir\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.920842 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.920798    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-model-cache\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.920842 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.920827    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-home\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.923006 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.922983    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-tls-certs\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.923104 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.923013    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-dshm\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.929553 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.929535    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4fc\" (UniqueName: \"kubernetes.io/projected/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-kube-api-access-tz4fc\") pod \"e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") " pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:46.981987 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:46.981961    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:47.273059 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:47.273035    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"]
Sep 09 04:36:47.278407 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:36:47.278367    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bf375db_4dd0_4ff9_86db_2ef63ed45d4e.slice/crio-4235850e81fbd7f109a155ee82dbde6c94bb5075698c0510d5e53db6a0280614 WatchSource:0}: Error finding container 4235850e81fbd7f109a155ee82dbde6c94bb5075698c0510d5e53db6a0280614: Status 404 returned error can't find the container with id 4235850e81fbd7f109a155ee82dbde6c94bb5075698c0510d5e53db6a0280614
Sep 09 04:36:47.428208 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:47.428169    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx" event={"ID":"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e","Type":"ContainerStarted","Data":"9eba59f9f19affc3e58276092fa18a1abd37e5ebb23546a9190611befcf51e4e"}
Sep 09 04:36:47.428400 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:47.428215    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx" event={"ID":"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e","Type":"ContainerStarted","Data":"4235850e81fbd7f109a155ee82dbde6c94bb5075698c0510d5e53db6a0280614"}
Sep 09 04:36:47.449746 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:47.449700    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx" podStartSLOduration=1.449684205 podStartE2EDuration="1.449684205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:36:47.447909778 +0000 UTC m=+2306.974280497" watchObservedRunningTime="2026-09-09 04:36:47.449684205 +0000 UTC m=+2306.976054944"
Sep 09 04:36:54.779867 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:54.779834    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:36:56.982615 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:56.982582    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:56.988920 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:56.988895    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:56.995271 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:56.995254    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:36:57.488924 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:36:57.488892    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:37:00.847865 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:00.847835    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"]
Sep 09 04:37:00.851391 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:00.851372    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:00.853957 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:00.853939    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2d97db68-kserve-self-signed-certs\""
Sep 09 04:37:00.863368 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:00.863344    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"]
Sep 09 04:37:00.883652 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:00.883627    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:37:00.929503 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:00.929478    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-dshm\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:00.929606 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:00.929509    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-tmp-dir\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:00.929652 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:00.929606    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-model-cache\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:00.929652 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:00.929643    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfxd\" (UniqueName: \"kubernetes.io/projected/e06c0767-df56-4a11-8ea3-6c8e367dd244-kube-api-access-slfxd\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:00.929730 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:00.929671    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-home\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:00.929774 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:00.929745    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06c0767-df56-4a11-8ea3-6c8e367dd244-tls-certs\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:01.030486 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.030437    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-dshm\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:01.030486 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.030489    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-tmp-dir\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:01.030693 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.030571    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-model-cache\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:01.030693 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.030622    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfxd\" (UniqueName: \"kubernetes.io/projected/e06c0767-df56-4a11-8ea3-6c8e367dd244-kube-api-access-slfxd\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:01.030693 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.030686    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-home\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:01.030865 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.030747    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06c0767-df56-4a11-8ea3-6c8e367dd244-tls-certs\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:01.030964 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.030943    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-tmp-dir\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:01.031133 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.031100    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-home\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:01.031247 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.031217    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-model-cache\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:01.032800 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.032778    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-dshm\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:01.033293 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.033273    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06c0767-df56-4a11-8ea3-6c8e367dd244-tls-certs\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:01.040143 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.040125    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfxd\" (UniqueName: \"kubernetes.io/projected/e06c0767-df56-4a11-8ea3-6c8e367dd244-kube-api-access-slfxd\") pod \"e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") " pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:01.161016 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.160947    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:01.435517 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.435493    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"]
Sep 09 04:37:01.437006 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:37:01.436980    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode06c0767_df56_4a11_8ea3_6c8e367dd244.slice/crio-b1a48551faeb3ef5434786cdb040df0330b9af630b1e6885e974b5debe048323 WatchSource:0}: Error finding container b1a48551faeb3ef5434786cdb040df0330b9af630b1e6885e974b5debe048323: Status 404 returned error can't find the container with id b1a48551faeb3ef5434786cdb040df0330b9af630b1e6885e974b5debe048323
Sep 09 04:37:01.500070 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:01.500047    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw" event={"ID":"e06c0767-df56-4a11-8ea3-6c8e367dd244","Type":"ContainerStarted","Data":"b1a48551faeb3ef5434786cdb040df0330b9af630b1e6885e974b5debe048323"}
Sep 09 04:37:02.506325 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:02.506286    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw" event={"ID":"e06c0767-df56-4a11-8ea3-6c8e367dd244","Type":"ContainerStarted","Data":"7674d23bc297805cdfc5560eee68e11eae747e859d494fb10018902205f15d4d"}
Sep 09 04:37:02.527721 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:02.527673    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw" podStartSLOduration=2.527659205 podStartE2EDuration="2.527659205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:37:02.526089093 +0000 UTC m=+2322.052459803" watchObservedRunningTime="2026-09-09 04:37:02.527659205 +0000 UTC m=+2322.054029915"
Sep 09 04:37:03.083116 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:03.083083    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:37:11.161379 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:11.161345    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:11.161868 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:11.161399    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:11.173550 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:11.173531    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:11.568375 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:11.568347    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:15.480880 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:15.480798    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:37:37.837511 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:37.837477    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"]
Sep 09 04:37:37.845852 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:37.845801    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:37:39.632699 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:39.632666    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"]
Sep 09 04:37:39.652637 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:39.652604    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:37:40.458471 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:40.458419    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:37:48.965211 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:48.965176    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:37:52.857380 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:52.857323    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx" podUID="9bf375db-4dd0-4ff9-86db-2ef63ed45d4e" containerName="main" containerID="cri-o://9eba59f9f19affc3e58276092fa18a1abd37e5ebb23546a9190611befcf51e4e" gracePeriod=45
Sep 09 04:37:53.115245 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.115187    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:37:53.164703 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.164672    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-home\") pod \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") "
Sep 09 04:37:53.164900 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.164764    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz4fc\" (UniqueName: \"kubernetes.io/projected/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-kube-api-access-tz4fc\") pod \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") "
Sep 09 04:37:53.164900 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.164791    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-dshm\") pod \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") "
Sep 09 04:37:53.164900 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.164838    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-tmp-dir\") pod \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") "
Sep 09 04:37:53.165072 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.165001    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-home" (OuterVolumeSpecName: "home") pod "9bf375db-4dd0-4ff9-86db-2ef63ed45d4e" (UID: "9bf375db-4dd0-4ff9-86db-2ef63ed45d4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:37:53.165136 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.165101    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-model-cache\") pod \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") "
Sep 09 04:37:53.165268 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.165210    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-tls-certs\") pod \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\" (UID: \"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e\") "
Sep 09 04:37:53.165268 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.165099    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9bf375db-4dd0-4ff9-86db-2ef63ed45d4e" (UID: "9bf375db-4dd0-4ff9-86db-2ef63ed45d4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:37:53.165510 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.165275    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "9bf375db-4dd0-4ff9-86db-2ef63ed45d4e" (UID: "9bf375db-4dd0-4ff9-86db-2ef63ed45d4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:37:53.165666 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.165646    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:37:53.165750 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.165675    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:37:53.165750 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.165691    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:37:53.167957 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.167856    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-kube-api-access-tz4fc" (OuterVolumeSpecName: "kube-api-access-tz4fc") pod "9bf375db-4dd0-4ff9-86db-2ef63ed45d4e" (UID: "9bf375db-4dd0-4ff9-86db-2ef63ed45d4e"). InnerVolumeSpecName "kube-api-access-tz4fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:37:53.167957 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.167921    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-dshm" (OuterVolumeSpecName: "dshm") pod "9bf375db-4dd0-4ff9-86db-2ef63ed45d4e" (UID: "9bf375db-4dd0-4ff9-86db-2ef63ed45d4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:37:53.167957 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.167929    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bf375db-4dd0-4ff9-86db-2ef63ed45d4e" (UID: "9bf375db-4dd0-4ff9-86db-2ef63ed45d4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:37:53.266270 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.266245    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz4fc\" (UniqueName: \"kubernetes.io/projected/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-kube-api-access-tz4fc\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:37:53.266270 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.266270    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:37:53.266427 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.266283    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:37:53.774586 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.774551    2553 generic.go:356] "Generic (PLEG): container finished" podID="9bf375db-4dd0-4ff9-86db-2ef63ed45d4e" containerID="9eba59f9f19affc3e58276092fa18a1abd37e5ebb23546a9190611befcf51e4e" exitCode=0
Sep 09 04:37:53.774759 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.774610    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"
Sep 09 04:37:53.774759 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.774646    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx" event={"ID":"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e","Type":"ContainerDied","Data":"9eba59f9f19affc3e58276092fa18a1abd37e5ebb23546a9190611befcf51e4e"}
Sep 09 04:37:53.774759 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.774692    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx" event={"ID":"9bf375db-4dd0-4ff9-86db-2ef63ed45d4e","Type":"ContainerDied","Data":"4235850e81fbd7f109a155ee82dbde6c94bb5075698c0510d5e53db6a0280614"}
Sep 09 04:37:53.774759 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.774709    2553 scope.go:117] "RemoveContainer" containerID="9eba59f9f19affc3e58276092fa18a1abd37e5ebb23546a9190611befcf51e4e"
Sep 09 04:37:53.783764 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.783746    2553 scope.go:117] "RemoveContainer" containerID="9eba59f9f19affc3e58276092fa18a1abd37e5ebb23546a9190611befcf51e4e"
Sep 09 04:37:53.784025 ip-10-0-135-219 kubenswrapper[2553]: E0909 04:37:53.784009    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eba59f9f19affc3e58276092fa18a1abd37e5ebb23546a9190611befcf51e4e\": container with ID starting with 9eba59f9f19affc3e58276092fa18a1abd37e5ebb23546a9190611befcf51e4e not found: ID does not exist" containerID="9eba59f9f19affc3e58276092fa18a1abd37e5ebb23546a9190611befcf51e4e"
Sep 09 04:37:53.784097 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.784033    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eba59f9f19affc3e58276092fa18a1abd37e5ebb23546a9190611befcf51e4e"} err="failed to get container status \"9eba59f9f19affc3e58276092fa18a1abd37e5ebb23546a9190611befcf51e4e\": rpc error: code = NotFound desc = could not find container \"9eba59f9f19affc3e58276092fa18a1abd37e5ebb23546a9190611befcf51e4e\": container with ID starting with 9eba59f9f19affc3e58276092fa18a1abd37e5ebb23546a9190611befcf51e4e not found: ID does not exist"
Sep 09 04:37:53.796266 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.796246    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"]
Sep 09 04:37:53.800560 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:53.800540    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2d97db68-kserve-b6b44554c-8j6qx"]
Sep 09 04:37:54.652512 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.652471    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw" podUID="e06c0767-df56-4a11-8ea3-6c8e367dd244" containerName="main" containerID="cri-o://7674d23bc297805cdfc5560eee68e11eae747e859d494fb10018902205f15d4d" gracePeriod=45
Sep 09 04:37:54.781582 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.781555    2553 generic.go:356] "Generic (PLEG): container finished" podID="e06c0767-df56-4a11-8ea3-6c8e367dd244" containerID="7674d23bc297805cdfc5560eee68e11eae747e859d494fb10018902205f15d4d" exitCode=0
Sep 09 04:37:54.781701 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.781631    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw" event={"ID":"e06c0767-df56-4a11-8ea3-6c8e367dd244","Type":"ContainerDied","Data":"7674d23bc297805cdfc5560eee68e11eae747e859d494fb10018902205f15d4d"}
Sep 09 04:37:54.903432 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.903383    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:54.980546 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.980519    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06c0767-df56-4a11-8ea3-6c8e367dd244-tls-certs\") pod \"e06c0767-df56-4a11-8ea3-6c8e367dd244\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") "
Sep 09 04:37:54.980698 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.980584    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-tmp-dir\") pod \"e06c0767-df56-4a11-8ea3-6c8e367dd244\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") "
Sep 09 04:37:54.980698 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.980631    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-model-cache\") pod \"e06c0767-df56-4a11-8ea3-6c8e367dd244\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") "
Sep 09 04:37:54.980698 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.980670    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-dshm\") pod \"e06c0767-df56-4a11-8ea3-6c8e367dd244\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") "
Sep 09 04:37:54.980895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.980766    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-home\") pod \"e06c0767-df56-4a11-8ea3-6c8e367dd244\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") "
Sep 09 04:37:54.980895 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.980817    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slfxd\" (UniqueName: \"kubernetes.io/projected/e06c0767-df56-4a11-8ea3-6c8e367dd244-kube-api-access-slfxd\") pod \"e06c0767-df56-4a11-8ea3-6c8e367dd244\" (UID: \"e06c0767-df56-4a11-8ea3-6c8e367dd244\") "
Sep 09 04:37:54.981064 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.981038    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-model-cache" (OuterVolumeSpecName: "model-cache") pod "e06c0767-df56-4a11-8ea3-6c8e367dd244" (UID: "e06c0767-df56-4a11-8ea3-6c8e367dd244"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:37:54.981138 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.981043    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e06c0767-df56-4a11-8ea3-6c8e367dd244" (UID: "e06c0767-df56-4a11-8ea3-6c8e367dd244"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:37:54.981138 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.981064    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-home" (OuterVolumeSpecName: "home") pod "e06c0767-df56-4a11-8ea3-6c8e367dd244" (UID: "e06c0767-df56-4a11-8ea3-6c8e367dd244"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:37:54.981235 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.981213    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-tmp-dir\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:37:54.981235 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.981227    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-model-cache\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:37:54.981356 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.981236    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-home\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:37:54.982825 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.982802    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e06c0767-df56-4a11-8ea3-6c8e367dd244-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e06c0767-df56-4a11-8ea3-6c8e367dd244" (UID: "e06c0767-df56-4a11-8ea3-6c8e367dd244"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:37:54.983038 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.982988    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-dshm" (OuterVolumeSpecName: "dshm") pod "e06c0767-df56-4a11-8ea3-6c8e367dd244" (UID: "e06c0767-df56-4a11-8ea3-6c8e367dd244"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:37:54.983385 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.983367    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06c0767-df56-4a11-8ea3-6c8e367dd244-kube-api-access-slfxd" (OuterVolumeSpecName: "kube-api-access-slfxd") pod "e06c0767-df56-4a11-8ea3-6c8e367dd244" (UID: "e06c0767-df56-4a11-8ea3-6c8e367dd244"). InnerVolumeSpecName "kube-api-access-slfxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:37:54.989709 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:54.989687    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bf375db-4dd0-4ff9-86db-2ef63ed45d4e" path="/var/lib/kubelet/pods/9bf375db-4dd0-4ff9-86db-2ef63ed45d4e/volumes"
Sep 09 04:37:55.081912 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:55.081881    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06c0767-df56-4a11-8ea3-6c8e367dd244-dshm\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:37:55.081912 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:55.081913    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slfxd\" (UniqueName: \"kubernetes.io/projected/e06c0767-df56-4a11-8ea3-6c8e367dd244-kube-api-access-slfxd\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:37:55.081912 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:55.081923    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06c0767-df56-4a11-8ea3-6c8e367dd244-tls-certs\") on node \"ip-10-0-135-219.ec2.internal\" DevicePath \"\""
Sep 09 04:37:55.759069 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:55.759043    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:37:55.788800 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:55.788775    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"
Sep 09 04:37:55.788944 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:55.788817    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw" event={"ID":"e06c0767-df56-4a11-8ea3-6c8e367dd244","Type":"ContainerDied","Data":"b1a48551faeb3ef5434786cdb040df0330b9af630b1e6885e974b5debe048323"}
Sep 09 04:37:55.788944 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:55.788854    2553 scope.go:117] "RemoveContainer" containerID="7674d23bc297805cdfc5560eee68e11eae747e859d494fb10018902205f15d4d"
Sep 09 04:37:55.806576 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:55.806551    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"]
Sep 09 04:37:55.810730 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:55.810711    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2d97db68-kserve-9d96fd9dc-62tfw"]
Sep 09 04:37:56.988777 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:37:56.988741    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e06c0767-df56-4a11-8ea3-6c8e367dd244" path="/var/lib/kubelet/pods/e06c0767-df56-4a11-8ea3-6c8e367dd244/volumes"
Sep 09 04:38:03.465723 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:03.465688    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:38:11.866738 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:11.866707    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:38:19.866893 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:19.866856    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:38:21.589809 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:21.589775    2553 scope.go:117] "RemoveContainer" containerID="fb037bde3ec8ca7e1c8d81d91417bc9b286b3c69963eb895308f8732ee956e8d"
Sep 09 04:38:21.599347 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:21.599325    2553 scope.go:117] "RemoveContainer" containerID="912c620145e5823c2b2ecb0e6e62b1074526e09f734ec19b2bff31df88d6d501"
Sep 09 04:38:21.607471 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:21.607430    2553 scope.go:117] "RemoveContainer" containerID="d4bb1aed5896e61051735327104b433e9685f9fe936ea6ec5da12b42a70c4eb2"
Sep 09 04:38:29.169474 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:29.169425    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:38:34.865454 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:34.865414    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rbp4j"]
Sep 09 04:38:44.038860 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:44.038790    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pq5qh_f2741325-8004-410d-9682-26b1e1f8b6cb/server/0.log"
Sep 09 04:38:44.410589 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:44.410515    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5996d6cf95-nx2mg_7cbb60ae-799c-4198-8e94-c5aed9cff219/copy-manifests/0.log"
Sep 09 04:38:44.418060 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:44.418035    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5996d6cf95-nx2mg_7cbb60ae-799c-4198-8e94-c5aed9cff219/manager/0.log"
Sep 09 04:38:45.254194 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:45.254165    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29815455-7shl5_6913ea88-2577-4ad2-844e-cc5ce75097b2/cleanup/0.log"
Sep 09 04:38:45.356365 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:45.356340    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29815470-kf77j_7150de37-2f88-4075-9104-048a537a21e2/cleanup/0.log"
Sep 09 04:38:45.464742 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:45.464718    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w4-85732c-29815455-z72hj_e4f44517-07ec-4d4c-8a7f-fbff9f306ccf/cleanup/0.log"
Sep 09 04:38:47.056634 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:47.056604    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-rbp4j_6272166e-49cb-4c5d-8aaf-fa87f71adf02/limitador/0.log"
Sep 09 04:38:47.533989 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:47.533957    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2_1eea3e85-da66-419b-9e8c-890e5a6afc6d/istio-proxy/0.log"
Sep 09 04:38:47.901963 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:47.901890    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7449ffbbd4-g5hqj_822e9f2c-cd73-42f1-9f9f-39f752bea97e/kube-auth-proxy/0.log"
Sep 09 04:38:48.132357 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:48.132330    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bfslf_f8f77855-b526-401c-b8a5-3ccf9360997c/istio-proxy/1.log"
Sep 09 04:38:49.293498 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:49.293472    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-scdgl_47db100f-4385-4d60-b496-1309ab3631ca/main/0.log"
Sep 09 04:38:49.513099 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:49.513072    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhhtj6_00abc0f1-4162-479b-b551-d728228711c0/main/0.log"
Sep 09 04:38:49.627256 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:49.627191    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-gfd5t_605df143-78f1-4424-ba50-b28c94bb0722/main/0.log"
Sep 09 04:38:49.745376 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:49.745356    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-sdxz8_cd55cd03-51e6-4481-9ba9-bd656969ca62/main/0.log"
Sep 09 04:38:55.580428 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:55.580393    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:38:56.815872 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:56.815840    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:38:58.847722 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:58.847695    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kpwhr_a7d64662-bc9c-4fd3-b0ee-0f4b2e21e3db/global-pull-secret-syncer/0.log"
Sep 09 04:38:58.897671 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:58.897647    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log"
Sep 09 04:38:58.983758 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:38:58.983733    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-219.ec2.internal_5cbec884f0899e64243f4816842f939e/haproxy/0.log"
Sep 09 04:39:02.046459 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:02.046410    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sc28_d0479cc6-6713-4586-afc7-86451c83d955/konnectivity-agent/0.log" limit=1024
Sep 09 04:39:03.971624 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:03.971593    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-rbp4j_6272166e-49cb-4c5d-8aaf-fa87f71adf02/limitador/0.log"
Sep 09 04:39:05.692097 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:05.692064    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_956fd919-b132-466b-9ddf-451c34ab9a22/alertmanager/0.log"
Sep 09 04:39:05.720258 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:05.720236    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_956fd919-b132-466b-9ddf-451c34ab9a22/config-reloader/0.log"
Sep 09 04:39:05.742899 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:05.742876    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_956fd919-b132-466b-9ddf-451c34ab9a22/kube-rbac-proxy-web/0.log"
Sep 09 04:39:05.768040 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:05.767990    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_956fd919-b132-466b-9ddf-451c34ab9a22/kube-rbac-proxy/0.log"
Sep 09 04:39:05.793125 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:05.793110    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_956fd919-b132-466b-9ddf-451c34ab9a22/kube-rbac-proxy-metric/0.log"
Sep 09 04:39:05.818628 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:05.818608    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_956fd919-b132-466b-9ddf-451c34ab9a22/prom-label-proxy/0.log"
Sep 09 04:39:05.848058 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:05.848042    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_956fd919-b132-466b-9ddf-451c34ab9a22/init-config-reloader/0.log"
Sep 09 04:39:05.954734 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:05.954679    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-dpccj_317b9d62-0af8-4984-a8bf-920ffa946469/kube-state-metrics/0.log"
Sep 09 04:39:05.975278 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:05.975264    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-dpccj_317b9d62-0af8-4984-a8bf-920ffa946469/kube-rbac-proxy-main/0.log"
Sep 09 04:39:06.003760 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.003725    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-dpccj_317b9d62-0af8-4984-a8bf-920ffa946469/kube-rbac-proxy-self/0.log"
Sep 09 04:39:06.031041 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.031024    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-69df5c8d65-28k72_662b2ae2-722e-4de5-87db-da25b85882d2/metrics-server/0.log"
Sep 09 04:39:06.059437 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.059418    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-rlbkn_84ebfd1d-c5ce-48af-ad63-9db8df98432c/monitoring-plugin/0.log"
Sep 09 04:39:06.097082 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.097067    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gdzk5_93ba5076-012e-4286-b6c2-8fdda7591135/node-exporter/0.log"
Sep 09 04:39:06.126620 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.126606    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gdzk5_93ba5076-012e-4286-b6c2-8fdda7591135/kube-rbac-proxy/0.log"
Sep 09 04:39:06.155467 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.155434    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gdzk5_93ba5076-012e-4286-b6c2-8fdda7591135/init-textfile/0.log"
Sep 09 04:39:06.376013 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.375991    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-qwkzn_30fe4953-3b75-473f-a4ec-ca7ff55ba07d/kube-rbac-proxy-main/0.log"
Sep 09 04:39:06.399395 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.399373    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-qwkzn_30fe4953-3b75-473f-a4ec-ca7ff55ba07d/kube-rbac-proxy-self/0.log"
Sep 09 04:39:06.423267 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.423244    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-qwkzn_30fe4953-3b75-473f-a4ec-ca7ff55ba07d/openshift-state-metrics/0.log"
Sep 09 04:39:06.472131 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.472113    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd6d0785-4895-40ce-a916-5ca5c7ba6101/prometheus/0.log"
Sep 09 04:39:06.499075 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.499050    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd6d0785-4895-40ce-a916-5ca5c7ba6101/config-reloader/0.log"
Sep 09 04:39:06.526633 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.526614    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd6d0785-4895-40ce-a916-5ca5c7ba6101/thanos-sidecar/0.log"
Sep 09 04:39:06.556476 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.556436    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd6d0785-4895-40ce-a916-5ca5c7ba6101/kube-rbac-proxy-web/0.log"
Sep 09 04:39:06.581109 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.581093    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd6d0785-4895-40ce-a916-5ca5c7ba6101/kube-rbac-proxy/0.log"
Sep 09 04:39:06.604022 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.604001    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd6d0785-4895-40ce-a916-5ca5c7ba6101/kube-rbac-proxy-thanos/0.log"
Sep 09 04:39:06.628997 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.628944    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd6d0785-4895-40ce-a916-5ca5c7ba6101/init-config-reloader/0.log"
Sep 09 04:39:06.658723 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.658705    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-ndc25_2aec475f-c9df-4064-a147-04f43d2f50e9/prometheus-operator/0.log"
Sep 09 04:39:06.679977 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.679961    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-ndc25_2aec475f-c9df-4064-a147-04f43d2f50e9/kube-rbac-proxy/0.log"
Sep 09 04:39:06.743435 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.743414    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78695676b7-ppbh6_f726385e-a889-4833-8d54-40e3856eb95b/telemeter-client/0.log"
Sep 09 04:39:06.768183 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.768166    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78695676b7-ppbh6_f726385e-a889-4833-8d54-40e3856eb95b/reload/0.log"
Sep 09 04:39:06.791375 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.791359    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78695676b7-ppbh6_f726385e-a889-4833-8d54-40e3856eb95b/kube-rbac-proxy/0.log"
Sep 09 04:39:06.834822 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.834800    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f97f9cd-2qngm_aeaf7fa0-947d-4f08-b87c-344f6f366d40/thanos-query/0.log"
Sep 09 04:39:06.857990 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.857971    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f97f9cd-2qngm_aeaf7fa0-947d-4f08-b87c-344f6f366d40/kube-rbac-proxy-web/0.log"
Sep 09 04:39:06.882870 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.882820    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f97f9cd-2qngm_aeaf7fa0-947d-4f08-b87c-344f6f366d40/kube-rbac-proxy/0.log"
Sep 09 04:39:06.904727 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.904703    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f97f9cd-2qngm_aeaf7fa0-947d-4f08-b87c-344f6f366d40/prom-label-proxy/0.log"
Sep 09 04:39:06.926901 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.926882    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f97f9cd-2qngm_aeaf7fa0-947d-4f08-b87c-344f6f366d40/kube-rbac-proxy-rules/0.log"
Sep 09 04:39:06.949240 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:06.949222    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f97f9cd-2qngm_aeaf7fa0-947d-4f08-b87c-344f6f366d40/kube-rbac-proxy-metrics/0.log"
Sep 09 04:39:09.292917 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.292888    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"]
Sep 09 04:39:09.293548 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.293516    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e06c0767-df56-4a11-8ea3-6c8e367dd244" containerName="main"
Sep 09 04:39:09.293548 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.293536    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e06c0767-df56-4a11-8ea3-6c8e367dd244" containerName="main"
Sep 09 04:39:09.293548 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.293552    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bf375db-4dd0-4ff9-86db-2ef63ed45d4e" containerName="main"
Sep 09 04:39:09.293808 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.293561    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bf375db-4dd0-4ff9-86db-2ef63ed45d4e" containerName="main"
Sep 09 04:39:09.293808 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.293649    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bf375db-4dd0-4ff9-86db-2ef63ed45d4e" containerName="main"
Sep 09 04:39:09.293808 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.293658    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e06c0767-df56-4a11-8ea3-6c8e367dd244" containerName="main"
Sep 09 04:39:09.296947 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.296930    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.299182 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.299155    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-smpj7\"/\"openshift-service-ca.crt\""
Sep 09 04:39:09.299182 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.299169    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-smpj7\"/\"default-dockercfg-s57v2\""
Sep 09 04:39:09.300108 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.300090    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-smpj7\"/\"kube-root-ca.crt\""
Sep 09 04:39:09.306293 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.306272    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"]
Sep 09 04:39:09.398625 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.398597    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-lib-modules\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.398735 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.398632    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-podres\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.398735 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.398657    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-proc\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.398808 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.398738    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-sys\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.398808 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.398765    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ssj\" (UniqueName: \"kubernetes.io/projected/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-kube-api-access-d6ssj\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.500148 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.500125    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-lib-modules\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.500261 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.500152    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-podres\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.500261 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.500170    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-proc\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.500261 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.500189    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-sys\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.500261 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.500208    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ssj\" (UniqueName: \"kubernetes.io/projected/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-kube-api-access-d6ssj\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.500502 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.500268    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-proc\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.500502 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.500282    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-podres\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.500502 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.500284    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-sys\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.500502 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.500298    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-lib-modules\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.508791 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.508772    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ssj\" (UniqueName: \"kubernetes.io/projected/bbe1ea02-568a-40db-ae3c-33a9bba8cc75-kube-api-access-d6ssj\") pod \"perf-node-gather-daemonset-r6s2w\" (UID: \"bbe1ea02-568a-40db-ae3c-33a9bba8cc75\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.607729 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.607667    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:09.879002 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:09.878975    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"]
Sep 09 04:39:09.881350 ip-10-0-135-219 kubenswrapper[2553]: W0909 04:39:09.881315    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbbe1ea02_568a_40db_ae3c_33a9bba8cc75.slice/crio-ea0499538ed769788892997c78e25231cd949f89d46adb0d29f08a4165dbfb61 WatchSource:0}: Error finding container ea0499538ed769788892997c78e25231cd949f89d46adb0d29f08a4165dbfb61: Status 404 returned error can't find the container with id ea0499538ed769788892997c78e25231cd949f89d46adb0d29f08a4165dbfb61
Sep 09 04:39:10.158259 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:10.158184    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w" event={"ID":"bbe1ea02-568a-40db-ae3c-33a9bba8cc75","Type":"ContainerStarted","Data":"dbde82fa297b5b5b0bd872798b6e50840297e1caed02064dfe011eb8c00d5af4"}
Sep 09 04:39:10.158259 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:10.158251    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w" event={"ID":"bbe1ea02-568a-40db-ae3c-33a9bba8cc75","Type":"ContainerStarted","Data":"ea0499538ed769788892997c78e25231cd949f89d46adb0d29f08a4165dbfb61"}
Sep 09 04:39:10.158410 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:10.158380    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:10.176933 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:10.176893    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w" podStartSLOduration=1.176878665 podStartE2EDuration="1.176878665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 04:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:39:10.175008925 +0000 UTC m=+2449.701379644" watchObservedRunningTime="2026-09-09 04:39:10.176878665 +0000 UTC m=+2449.703249408"
Sep 09 04:39:11.562798 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:11.562759    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lggsm_5a976dc5-634b-425c-88f4-d8e2822e1911/dns/0.log"
Sep 09 04:39:11.584308 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:11.584286    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lggsm_5a976dc5-634b-425c-88f4-d8e2822e1911/kube-rbac-proxy/0.log"
Sep 09 04:39:11.657583 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:11.657557    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-86q49_18092042-a374-4054-9dfa-6b0a511d7c0f/dns-node-resolver/0.log"
Sep 09 04:39:12.316454 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:12.316420    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k29sf_67c61063-131e-4cc1-84ad-889cded7b195/node-ca/0.log"
Sep 09 04:39:13.389211 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:13.389181    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2_1eea3e85-da66-419b-9e8c-890e5a6afc6d/istio-proxy/0.log"
Sep 09 04:39:13.706616 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:13.706551    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7449ffbbd4-g5hqj_822e9f2c-cd73-42f1-9f9f-39f752bea97e/kube-auth-proxy/0.log"
Sep 09 04:39:13.769645 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:13.769602    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bfslf_f8f77855-b526-401c-b8a5-3ccf9360997c/istio-proxy/0.log"
Sep 09 04:39:13.777620 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:13.777598    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bfslf_f8f77855-b526-401c-b8a5-3ccf9360997c/istio-proxy/1.log"
Sep 09 04:39:14.880101 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:14.880069    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kdvjc_cfbd047c-3b5e-442f-938b-667a26e580f3/serve-healthcheck-canary/0.log"
Sep 09 04:39:15.528660 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:15.528632    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8d765_5e8503b7-f81e-4115-aa5d-e38becb6e75a/kube-rbac-proxy/0.log"
Sep 09 04:39:15.550216 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:15.550194    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8d765_5e8503b7-f81e-4115-aa5d-e38becb6e75a/exporter/0.log"
Sep 09 04:39:15.572622 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:15.572583    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8d765_5e8503b7-f81e-4115-aa5d-e38becb6e75a/extractor/0.log"
Sep 09 04:39:16.173845 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:16.173820    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-r6s2w"
Sep 09 04:39:18.176625 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:18.176596    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pq5qh_f2741325-8004-410d-9682-26b1e1f8b6cb/server/0.log"
Sep 09 04:39:18.335114 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:18.335083    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5996d6cf95-nx2mg_7cbb60ae-799c-4198-8e94-c5aed9cff219/manager/0.log"
Sep 09 04:39:18.357032 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:18.357012    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5996d6cf95-nx2mg_7cbb60ae-799c-4198-8e94-c5aed9cff219/copy-manifests/0.log"
Sep 09 04:39:19.774934 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:19.774909    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2_1eea3e85-da66-419b-9e8c-890e5a6afc6d/istio-proxy/0.log"
Sep 09 04:39:19.929879 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:19.929853    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7486b797d6-b6ggb_74b48f4d-caa8-40d5-9f9d-0a60e4d8651e/manager/0.log"
Sep 09 04:39:20.074348 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:20.074276    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7449ffbbd4-g5hqj_822e9f2c-cd73-42f1-9f9f-39f752bea97e/kube-auth-proxy/0.log"
Sep 09 04:39:20.127080 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:20.127058    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bfslf_f8f77855-b526-401c-b8a5-3ccf9360997c/istio-proxy/0.log"
Sep 09 04:39:20.135174 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:20.135156    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bfslf_f8f77855-b526-401c-b8a5-3ccf9360997c/istio-proxy/1.log"
Sep 09 04:39:21.846864 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:21.846793    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xt7n2_1eea3e85-da66-419b-9e8c-890e5a6afc6d/istio-proxy/0.log"
Sep 09 04:39:22.149314 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:22.149238    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7449ffbbd4-g5hqj_822e9f2c-cd73-42f1-9f9f-39f752bea97e/kube-auth-proxy/0.log"
Sep 09 04:39:22.191204 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:22.191178    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bfslf_f8f77855-b526-401c-b8a5-3ccf9360997c/istio-proxy/0.log"
Sep 09 04:39:22.199234 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:22.199216    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bfslf_f8f77855-b526-401c-b8a5-3ccf9360997c/istio-proxy/1.log"
Sep 09 04:39:27.116127 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:27.116096    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pj8mr_b5b1820e-2357-48d0-ad83-f9859f396e8d/kube-multus-additional-cni-plugins/0.log"
Sep 09 04:39:27.139106 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:27.139081    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pj8mr_b5b1820e-2357-48d0-ad83-f9859f396e8d/egress-router-binary-copy/0.log"
Sep 09 04:39:27.161181 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:27.161162    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pj8mr_b5b1820e-2357-48d0-ad83-f9859f396e8d/cni-plugins/0.log"
Sep 09 04:39:27.183105 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:27.183086    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pj8mr_b5b1820e-2357-48d0-ad83-f9859f396e8d/bond-cni-plugin/0.log"
Sep 09 04:39:27.205097 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:27.205077    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pj8mr_b5b1820e-2357-48d0-ad83-f9859f396e8d/routeoverride-cni/0.log"
Sep 09 04:39:27.227150 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:27.227134    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pj8mr_b5b1820e-2357-48d0-ad83-f9859f396e8d/whereabouts-cni-bincopy/0.log"
Sep 09 04:39:27.249224 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:27.249200    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pj8mr_b5b1820e-2357-48d0-ad83-f9859f396e8d/whereabouts-cni/0.log"
Sep 09 04:39:27.537882 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:27.537856    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-z5ftt_e85290a3-6d41-402c-b0bf-2a6f18e41c72/kube-multus/0.log"
Sep 09 04:39:27.563437 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:27.563420    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4x7vw_9579406a-2f97-456a-8d1a-bb8db3d24ab6/network-metrics-daemon/0.log"
Sep 09 04:39:27.585202 ip-10-0-135-219 kubenswrapper[2553]: I0909 04:39:27.585188    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4x7vw_9579406a-2f97-456a-8d1a-bb8db3d24ab6/kube-rbac-proxy/0.log"